[jira] [Work logged] (BEAM-4798) IndexOutOfBoundsException when Flink parallelism > 1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4798?focusedWorklogId=133454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133454
 ]

ASF GitHub Bot logged work on BEAM-4798:


Author: ASF GitHub Bot
Created on: 10/Aug/18 04:19
Start Date: 10/Aug/18 04:19
Worklog Time Spent: 10m 
  Work Description: aljoscha commented on issue #6177: [BEAM-4798] Fix 
IndexOutOfBoundsException in Flink runner
URL: https://github.com/apache/beam/pull/6177#issuecomment-411970715
 
 
   Run Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133454)
Time Spent: 4h  (was: 3h 50m)

> IndexOutOfBoundsException when Flink parallelism > 1
> 
>
> Key: BEAM-4798
> URL: https://issues.apache.org/jira/browse/BEAM-4798
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Running job on Flink in streaming mode and get data from a Kafka topic with 
> parallelism > 1 causes an exception:
> {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:657)
>   at java.util.ArrayList.get(ArrayList.java:433)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:277)
>   at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
>   at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
>   at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
>   at 
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}
> It happens when number of Kafka topic partitions is less than value of 
> parallelism (number of task slots).
> So, workaround for now can be to set parallelism <= number of topic 
> partitions, thus if parallelism=2 then number_partitions >= 2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1220

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #6199 from udim/precommit-timeout

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fe0011514ef20b6cd276360819053a25c3bb26a6
Merge: f0fb7f6 4c47d87
Author: Charles Chen 
AuthorDate: Thu Aug 9 19:16:01 2018 -0700

Merge pull request #6199 from udim/precommit-timeout

Set precommit job timeouts to 30 minutes.

 .test-infra/jenkins/PrecommitJobBuilder.groovy | 2 +-
 .test-infra/jenkins/job_PreCommit_Go.groovy| 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

[beam] branch master updated (f0fb7f6 -> fe00115)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f0fb7f6  [BEAM-4846] Fixing updateOfflineRepository task.  (#6196)
 add 4c47d87  Set precommit job timeouts to 30 minutes.
 new fe00115  Merge pull request #6199 from udim/precommit-timeout

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/PrecommitJobBuilder.groovy | 2 +-
 .test-infra/jenkins/job_PreCommit_Go.groovy| 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1219

[Apache Jenkins Server]

See 


--
[...truncated 20.71 MB...]
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.131Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.178Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.447Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.494Z: Elided trivial flatten 
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.542Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.587Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.635Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.679Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.716Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.764Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.810Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.855Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Aug 10, 2018 12:50:30 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T00:50:27.892Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PreCommit_Java_Cron #207

[Apache Jenkins Server]

See 


Changes:

[aromanenko.dev] Added maxPublishBatchSize parameter to PubsubBoundedWriter 
class.

[aromanenko.dev] updated BoundedPubsubWriter to dynamically flush if queued 
messages

[aromanenko.dev] updated UnboundedPubsubSink to accept new parameters.

[aromanenko.dev] Resolve merging conflicts

[carlm] set maximum batch size to 10mb (10 * 1024 * 1024)

[lcwik] [BEAM-4658] Follow up on PR comments from #6050

[github] [BEAM-4846] Fixing updateOfflineRepository task.  (#6196)

--
[...truncated 12.93 MB...]
Aug 10, 2018 12:11:57 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..2=[{inputs}], f_int=[$t0], f_string=[$t2])
  BeamUnnestRel(correlation=[$cor0], joinType=[inner], 
requiredColumns=[{1}])
BeamIOSourceRel(table=[[beam, main]])
BeamUncollectRel
  BeamCalcRel(expr#0=[{inputs}], expr#1=[$cor0], 
expr#2=[$t1.f_stringArr_1], f_stringArr=[$t2])
BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testCardinality 
STANDARD_ERROR
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT CARDINALITY(`PCOLLECTION`.`f_stringArr`)
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[CARDINALITY($1)])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[CARDINALITY($t1)], EXPR$0=[$t2])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectRowsFromArrayOfRows STANDARD_ERROR
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_arrayOfRows=[$1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], f_arrayOfRows=[$t1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestLiteral 
STANDARD_ERROR
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `EXPR$0`.`EXPR$0`
FROM UNNEST(ARRAY['a', 'b', 'c'])
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[$0])
  Uncollect
LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
  LogicalValues(tuples=[[{ 0 }]])

Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], EXPR$0=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestNamedLiteral 
STANDARD_ERROR
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `t`.`f_string`
FROM UNNEST(ARRAY['a', 'b', 'c']) AS `t` (`f_string`)
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_string=[$0])
  LogicalProject(f_string=[$0])
Uncollect
  LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
LogicalValues(tuples=[[{ 0 }]])

Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], f_string=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectSingleRowFromArrayOfRows STANDARD_ERROR
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`[1]
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 10, 2018 12:11:58 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner 

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133402
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 10/Aug/18 00:05
Start Date: 10/Aug/18 00:05
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6185: [BEAM-5109] create 
python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411935369
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133402)
Time Spent: 3.5h  (was: 3h 20m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4744) Jars are overwritten during release with -Ppublishing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4744?focusedWorklogId=133399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133399
 ]

ASF GitHub Bot logged work on BEAM-4744:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:45
Start Date: 09/Aug/18 23:45
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5936: [BEAM-4744] Enable 
parallel publishing/release
URL: https://github.com/apache/beam/pull/5936#issuecomment-411932276
 
 
   I added a special `publish` property to JavaNatureConfiguration so that 
those examples project would not be published.
   
https://github.com/apache/beam/blob/f0fb7f67a54f815c9342adc87749ed0b7d962187/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L737
   
   What empty project was being published? I'm curious.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133399)
Time Spent: 1.5h  (was: 1h 20m)

> Jars are overwritten during release with -Ppublishing
> -
>
> Key: BEAM-4744
> URL: https://issues.apache.org/jira/browse/BEAM-4744
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Mark Liu
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.6.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> +cc [~pabloem] [~boyuanz] [~lukecwik]
> Few jars (*-shaded.jar and *-tests.jar) are missing if I run `./gradlew 
> -Ppublishing :publishToMavenLocal`. 
> However, I can get them through assemble and shadowJar. For example, 
> `./gradlew :beam-sdks-java-core:assemble :beam-sdks-java-core:shadowJar`.
> This could be the reason why we can't run parallel build during release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133398
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:42
Start Date: 09/Aug/18 23:42
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#discussion_r209094143
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 ##
 @@ -1170,4 +1171,32 @@ public void testTimestampMinusInterval() throws 
Exception {
 parseDate("1983-01-19 01:01:58"));
 checker.buildRunAndCheck();
   }
+
+  @Test
+  @SqlOperatorTest(name = "CASE", kind = "CASE")
+  @SqlOperatorTest(name = "NULLIF", kind = "NULLIF")
+  @SqlOperatorTest(name = "COALESCE", kind = "COALESCE")
+  public void testConditionalOperatorsAndFunctions() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END", "hello")
+.addExpr(
+"CASE 2 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"world")
+.addExpr(
+"CASE 3 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"bond")
+.addExpr("CASE " + "WHEN 1 = 1 THEN 'hello' " + "ELSE 'world' 
END", "hello")
+.addExpr("CASE " + "WHEN 1 > 1 THEN 'hello' " + "ELSE 'world' 
END", "world")
+.addExpr("NULLIF(5, 4) ", 5)
+.addExpr("NULLIF(4, 5) ", 4)
+.addExpr("NULLIF(5, 5)", null, FieldType.INT32)
 
 Review comment:
   It won't work neither because of 
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java#L156
   
   
   So `null` cannot call `.getClass` because of NullPointerException even we 
convert it. (Compiler does not buy this conversion). That's why I am passing in 
a FieldType.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133398)
Time Spent: 3h 50m  (was: 3h 40m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133396
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:40
Start Date: 09/Aug/18 23:40
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#issuecomment-411931452
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133396)
Time Spent: 3.5h  (was: 3h 20m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133397
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:40
Start Date: 09/Aug/18 23:40
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#issuecomment-411931452
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133397)
Time Spent: 3h 40m  (was: 3.5h)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133395
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:33
Start Date: 09/Aug/18 23:33
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6185: [BEAM-5109] create 
python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411930288
 
 
   R: @boyuanzz 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133395)
Time Spent: 3h 20m  (was: 3h 10m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133377
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:16
Start Date: 09/Aug/18 23:16
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index f3b5fb53134..bc22bad8abd 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -259,30 +259,6 @@ class BeamModulePlugin implements Plugin {
 // when attempting to resolve dependency issues.
 project.apply plugin: "project-report"
 
-// Apply a plugin which provides the 'updateOfflineRepository' task that 
creates an offline
-// repository. This offline repository satisfies all Gradle build 
dependencies and Java
-// project dependencies. The offline repository is placed within 
$rootDir/offline-repo
-// but can be overridden by specifying the 'offlineRepositoryRoot' Gradle 
option.
-// Note that parallel build must be disabled when executing 
'updateOfflineRepository'
-// by specifying '-Dorg.gradle.parallel=false', see
-// 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin/issues/3
-project.apply plugin: "io.pry.gradle.offline_dependencies"
-
-project.offlineDependencies {
-  repositories {
-maven { url offlineRepositoryRoot }
-mavenLocal()
-mavenCentral()
-jcenter()
-maven { url "https://plugins.gradle.org/m2/; }
-maven { url "http://repo.spring.io/plugins-release; }
-  }
-
-  includeSources = false
-  includeJavadocs = false
-  includeIvyXmls = false
-}
-
 /** 
***/
 // Define and export a map dependencies shared across multiple 
sub-projects.
 //
@@ -569,6 +545,28 @@ class BeamModulePlugin implements Plugin {
   }
   project.artifacts.archives project.packageTests
 
+  // Apply a plugin which provides the 'updateOfflineRepository' task that 
creates an offline
+  // repository. This offline repository satisfies all Gradle build 
dependencies and Java
+  // project dependencies. The offline repository is placed within 
$rootDir/offline-repo
+  // but can be overridden by specifying 
'-PofflineRepositoryRoot=/path/to/repo'.
+  // Note that parallel build must be disabled when executing 
'updateOfflineRepository'
+  // by specifying '--no-parallel', see
+  // 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin/issues/3
+  project.apply plugin: "io.pry.gradle.offline_dependencies"
+  project.offlineDependencies {
+repositories {
+  mavenLocal()
+  mavenCentral()
+  jcenter()
+  maven { url "https://plugins.gradle.org/m2/; }
+  maven { url "http://repo.spring.io/plugins-release; }
+  maven { url project.offlineRepositoryRoot }
+}
+includeSources = false
+includeJavadocs = false
+includeIvyXmls = false
+  }
+
   // Configures annotation processing for commonly used annotation 
processors
   // across all Java projects.
   project.apply plugin: "net.ltgt.apt"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133377)
Time Spent: 1.5h  (was: 1h 20m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> 

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133376
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:13
Start Date: 09/Aug/18 23:13
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411926860
 
 
   Well, it looks like the spotless check for buildSrc passed so feel free to 
merge this as I only made comment changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133376)
Time Spent: 1h 20m  (was: 1h 10m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> 

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133375
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:12
Start Date: 09/Aug/18 23:12
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411926713
 
 
   No clue : ) - but this is good enough to run `./gradlew 
updateOfflineRepository --no-parallel` and get all dependencies without issues.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133375)
Time Spent: 1h 10m  (was: 1h)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> 

[jira] [Assigned] (BEAM-4846) updateOfflineRepositoryRoot broken

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4846:
---

Assignee: Pablo Estrada

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at 
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at 
> 

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133373
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:10
Start Date: 09/Aug/18 23:10
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411926439
 
 
   Sweet, didn't think the fix was going to be this easy. Any clue as to why it 
wasn't working?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133373)
Time Spent: 1h  (was: 50m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133370=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133370
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:04
Start Date: 09/Aug/18 23:04
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411925343
 
 
   I only had to run PreCommit 8 times : D


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133370)
Time Spent: 50m  (was: 40m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133369
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 23:01
Start Date: 09/Aug/18 23:01
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6185: [BEAM-5109] create 
python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411924844
 
 
   Run Python Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133369)
Time Spent: 3h 10m  (was: 3h)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133367
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:57
Start Date: 09/Aug/18 22:57
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6185: [BEAM-5109] create 
python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411923995
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133367)
Time Spent: 3h  (was: 2h 50m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133368
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:57
Start Date: 09/Aug/18 22:57
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-411924063
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133368)
Time Spent: 5h 10m  (was: 5h)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133366
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:56
Start Date: 09/Aug/18 22:56
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6176: [[BEAM-5092] Row 
comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#issuecomment-411923832
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133366)
Time Spent: 7h 10m  (was: 7h)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1217

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5114) Create example uber jars for supported runners

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5114?focusedWorklogId=133353=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133353
 ]

ASF GitHub Bot logged work on BEAM-5114:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:26
Start Date: 09/Aug/18 22:26
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #6191: 
[BEAM-5114] Create example uber jars
URL: https://github.com/apache/beam/pull/6191#discussion_r209098073
 
 

 ##
 File path: examples/java/direct/build.gradle
 ##
 @@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+// Disable default shadow jar closure and include all class files and 
resources.
+applyJavaNature(shadowClosure: {})
 
 Review comment:
   Yeah, I'd like to challenge and ask why we wouldn't want these to be part of 
the release.  It would already be useful in my integration tests, and I'd 
imagine many beginner users would find a known-green fat jar to be useful for 
debugging their setups.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133353)
Time Spent: 1h 20m  (was: 1h 10m)

> Create example uber jars for supported runners
> --
>
> Key: BEAM-5114
> URL: https://issues.apache.org/jira/browse/BEAM-5114
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Producing these artifacts results in several benefits
>  * Gives an example of how to package user code for different runners
>  * Enables ad-hoc testing of runner changes against real user pipelines easier
>  * Enables integration testing end-to-end pipelines against different runner 
> services



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133347
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:23
Start Date: 09/Aug/18 22:23
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411917273
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133347)
Time Spent: 40m  (was: 0.5h)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133343
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:20
Start Date: 09/Aug/18 22:20
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#discussion_r209096749
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 ##
 @@ -152,12 +215,14 @@ private static ExpressionTestCase of(
 private transient List exps = new ArrayList<>();
 
 public ExpressionChecker addExpr(String expression, Object expectedValue) {
-  // Because of erasure, we can only automatically infer non-parameterized 
types
   TypeName resultTypeName = 
JAVA_CLASS_TO_TYPENAME.get(expectedValue.getClass());
   checkArgument(
   resultTypeName != null,
-  "Could not infer a Beam type for %s."
-  + " Parameterized types must be provided explicitly.");
+  String.format(
+  "The type of the expected object '%s' is unknown in 
'addExpr(String %s, Object %s)'"
 
 Review comment:
   Updated.
   
   I kept the first %s for now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133343)
Time Spent: 5h  (was: 4h 50m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4792) Add support for bounded SDF to all runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-4792:
---
Issue Type: New Feature  (was: Bug)

> Add support for bounded SDF to all runners
> --
>
> Key: BEAM-4792
> URL: https://issues.apache.org/jira/browse/BEAM-4792
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Executing bounded-per-element SDFs "naively" is pretty easy and doesn't 
> require any special runner features, because it can be expanded into regular 
> transforms. The naive implementation is sufficient for all runners except 
> Dataflow, which has liquid sharding: support for that is a separate JIRA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133340
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:19
Start Date: 09/Aug/18 22:19
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#discussion_r209096521
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 ##
 @@ -131,6 +192,8 @@ private static ExpressionTestCase of(
 abstract Object expectedResult();
 
 abstract FieldType resultFieldType();
+
+// abstract boolean isCompareNonExactNumber();
 
 Review comment:
   oops. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133340)
Time Spent: 4h 50m  (was: 4h 40m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=19=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-19
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:18
Start Date: 09/Aug/18 22:18
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209096201
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
 ##
 @@ -1425,6 +1425,7 @@ artifactId=${project.name}
 /** 
***/
 
 // Method to create the PortableValidatesRunnerTask.
+// Generated task expect :beam-sdks-java-core and :beam-runners-core-java 
to be in the class path.
 
 Review comment:
   Reword this to make it clear this is a gradle evaluation dependency (rather 
than a java classpath issue).
   
   Specifically, users must make sure the following is in the consuming 
module's `build.gradle` file:
   
   ```
   evaluationDependsOn(":beam-sdks-java-core")
   evaluationDependsOn(":beam-runners-core-java")
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 19)
Time Spent: 12h 10m  (was: 12h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=17=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-17
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:17
Start Date: 09/Aug/18 22:17
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#discussion_r209096193
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 ##
 @@ -1170,4 +1171,32 @@ public void testTimestampMinusInterval() throws 
Exception {
 parseDate("1983-01-19 01:01:58"));
 checker.buildRunAndCheck();
   }
+
+  @Test
+  @SqlOperatorTest(name = "CASE", kind = "CASE")
+  @SqlOperatorTest(name = "NULLIF", kind = "NULLIF")
+  @SqlOperatorTest(name = "COALESCE", kind = "COALESCE")
+  public void testConditionalOperatorsAndFunctions() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END", "hello")
+.addExpr(
+"CASE 2 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"world")
+.addExpr(
+"CASE 3 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"bond")
+.addExpr("CASE " + "WHEN 1 = 1 THEN 'hello' " + "ELSE 'world' 
END", "hello")
+.addExpr("CASE " + "WHEN 1 > 1 THEN 'hello' " + "ELSE 'world' 
END", "world")
+.addExpr("NULLIF(5, 4) ", 5)
+.addExpr("NULLIF(4, 5) ", 4)
+.addExpr("NULLIF(5, 5)", null, FieldType.INT32)
 
 Review comment:
   I could probably add a check for NULL before `expectedValue.getClass()` 
though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 17)
Time Spent: 3h 20m  (was: 3h 10m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4792) Add support for bounded SDF to all runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4792.

   Resolution: Fixed
Fix Version/s: 2.6.0

> Add support for bounded SDF to all runners
> --
>
> Key: BEAM-4792
> URL: https://issues.apache.org/jira/browse/BEAM-4792
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Executing bounded-per-element SDFs "naively" is pretty easy and doesn't 
> require any special runner features, because it can be expanded into regular 
> transforms. The naive implementation is sufficient for all runners except 
> Dataflow, which has liquid sharding: support for that is a separate JIRA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit cb6ac224c31033402fa5880cf24a9fba5c88bb2b
Author: Mergebot 
AuthorDate: Thu Aug 9 22:11:59 2018 +

Prepare repository for deployment.
---
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  | 28 +-
 2 files changed, 12 insertions(+), 18 deletions(-)

diff --git a/content/documentation/dsls/sql/overview/index.html 
b/content/documentation/dsls/sql/overview/index.html
index 2d0c8b4..fcd5879 100644
--- a/content/documentation/dsls/sql/overview/index.html
+++ b/content/documentation/dsls/sql/overview/index.html
@@ -256,7 +256,7 @@ big data processing with some streaming enhancements. 
Calcite provides the
 basic dialect underlying Beam SQL. We have added additional extensions to
 make it easy to leverage Beam’s unified batch/streaming model and support
 for complex data types.
-  BeamSql:
 
+  SqlTransform:
 
 the interface for creating PTransforms 
from SQL queries.
   Row:
 the type of elements that Beam SQL operates on. A PCollectionRow plays the role of a 
table.
diff --git a/content/documentation/dsls/sql/walkthrough/index.html 
b/content/documentation/dsls/sql/walkthrough/index.html
index 0b58348..0ba1520 100644
--- a/content/documentation/dsls/sql/walkthrough/index.html
+++ b/content/documentation/dsls/sql/walkthrough/index.html
@@ -224,7 +224,7 @@
 
 
   Row
-  BeamSql transform
+  SqlTransform
 
 
 
@@ -265,9 +265,9 @@ Types for more details on supported primitive data 
types.
   
 From in-memory data (typically for unit testing).
 
-Note: you have to explicitly specify the Row coder. In this example we’re doing it by 
calling Create.of(..).withCoder():
+Note: you have to explicitly specify the Row coder. In this example we’re doing it by 
calling Create.of(..):
 
-// Define the record type (i.e., 
schema).
+// Define the schema for the 
records.
 Schema appSchema = 
 Schema
   .builder()
@@ -293,10 +293,7 @@ Types for more details on supported primitive data 
types.
 
 
   
-  
-From a PCollectionT of records of some other 
type  (i.e.  T is not already a 
Row), by applying a ParDo that converts input records to Row format.
-
-Note: you have to manually set the coder of the result 
by calling setCoder(appSchema.getRowCoder()):
+  From a PCollectionT of records of some other 
type  (i.e.  T is not already a 
Row), by applying a ParDo that converts input records to Row format:
 // An example POJO class.
 class AppPojo {
   Integer appId;
@@ -330,21 +327,18 @@ Types for more details on supported primitive data 
types.
   // Output the Row representing the current 
POJO
   c.output(appRow);
 }
-  }))
-  .setCoder(appSchema.getRowCoder());
+  }));
 
 
   
-  
-As an output of another BeamSql query. Details in the next 
section.
-  
+  As an output of another SqlTransform. Details in the next 
section.
 
 
-Once you have a PCollectionRow in hand, you may use 
the BeamSql APIs to apply SQL queries to 
it.
+Once you have a PCollectionRow in hand, you may use 
SqlTransform to apply SQL queries to 
it.
 
-BeamSql transform
+SqlTransform
 
-BeamSql.query(queryString) method is 
the only API to create a PTransform
+SqlTransform.query(queryString) method is 
the only API to create a PTransform
 from a string representation of the SQL query. You can apply this PTransform
 to either a single PCollection or a 
PCollectionTuple which holds multiple
 PCollections:
@@ -352,7 +346,7 @@ to either a single PCollection or a 
   when applying to a single PCollection it can be referenced via the table 
name PCOLLECTION in the query:
 PCollectionRow 
filteredNames = testApps.apply(
-BeamSql.query(
+SqlTransform.query(
   "SELECT appId, description, rowtime "
 + "FROM PCOLLECTION "
 + "WHERE id=1"));
@@ -386,7 +380,7 @@ to either a single PCollection or a // and average rating per app 
 // by joining two PCollections
 PCollectionRow output 
= namesAndFoods.apply(
-BeamSql.query(
+SqlTransform.query(
 "SELECT Names.appId, COUNT(Reviews.rating), 
AVG(Reviews.rating)"
 + "FROM Apps INNER JOIN 
Reviews ON Apps.appId == Reviews.appId"));
 

[beam-site] branch asf-site updated (f5fde01 -> cb6ac22)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from f5fde01  Prepare repository for deployment.
 add 8b15877  Update SQL walkthrough for 2.6.0
 add 7ff1a24  This closes #529
 new cb6ac22  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  | 28 +-
 src/documentation/dsls/sql/overview.md |  2 +-
 src/documentation/dsls/sql/walkthrough.md  | 23 --
 4 files changed, 23 insertions(+), 32 deletions(-)

[beam-site] branch mergebot updated (585372b -> 7ff1a24)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 585372b  This closes #524
 add f5fde01  Prepare repository for deployment.
 new 8b15877  Update SQL walkthrough for 2.6.0
 new 7ff1a24  This closes #529

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/runners/apex/index.html | 34 ++-
 src/documentation/dsls/sql/overview.md|  2 +-
 src/documentation/dsls/sql/walkthrough.md | 23 --
 3 files changed, 29 insertions(+), 30 deletions(-)

[beam-site] 01/02: Update SQL walkthrough for 2.6.0

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8b158779271a92295c29270f1e3f8968a3b3cc40
Author: Anton Kedin 
AuthorDate: Tue Aug 7 15:21:52 2018 -0700

Update SQL walkthrough for 2.6.0
---
 src/documentation/dsls/sql/overview.md|  2 +-
 src/documentation/dsls/sql/walkthrough.md | 23 ++-
 2 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/src/documentation/dsls/sql/overview.md 
b/src/documentation/dsls/sql/overview.md
index 8a7ac29..7063b16 100644
--- a/src/documentation/dsls/sql/overview.md
+++ b/src/documentation/dsls/sql/overview.md
@@ -32,7 +32,7 @@ There are three main things you will need to know to use SQL 
in your pipeline:
basic dialect underlying Beam SQL. We have added additional extensions to
make it easy to leverage Beam's unified batch/streaming model and support
for complex data types.
- - [BeamSql]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/BeamSql.html): 
+ - [SqlTransform]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html): 
the interface for creating `PTransforms` from SQL queries.
  - [Row]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/values/Row.html):
the type of elements that Beam SQL operates on. A `PCollection` plays 
the role of a table.
diff --git a/src/documentation/dsls/sql/walkthrough.md 
b/src/documentation/dsls/sql/walkthrough.md
index 26f1ac1..57fa8fb 100644
--- a/src/documentation/dsls/sql/walkthrough.md
+++ b/src/documentation/dsls/sql/walkthrough.md
@@ -40,10 +40,10 @@ A `PCollection` can be obtained multiple ways, for 
example:
 
   - **From in-memory data** (typically for unit testing).
 
-**Note:** you have to explicitly specify the `Row` coder. In this example 
we're doing it by calling `Create.of(..).withCoder()`:
+**Note:** you have to explicitly specify the `Row` coder. In this example 
we're doing it by calling `Create.of(..)`:
 
 ```java
-// Define the record type (i.e., schema).
+// Define the schema for the records.
 Schema appSchema = 
 Schema
   .builder()
@@ -67,9 +67,7 @@ A `PCollection` can be obtained multiple ways, for 
example:
 .of(row)
 .withCoder(appSchema.getRowCoder()));
 ```
-  - **From a `PCollection` of records of some other type**  (i.e.  `T` is 
not already a `Row`), by applying a `ParDo` that converts input records to 
`Row` format.
-
-**Note:** you have to manually set the coder of the result by calling 
`setCoder(appSchema.getRowCoder())`:
+  - **From a `PCollection` of records of some other type**  (i.e.  `T` is 
not already a `Row`), by applying a `ParDo` that converts input records to 
`Row` format:
 ```java
 // An example POJO class.
 class AppPojo {
@@ -104,17 +102,16 @@ A `PCollection` can be obtained multiple ways, for 
example:
   // Output the Row representing the current POJO
   c.output(appRow);
 }
-  }))
-  .setCoder(appSchema.getRowCoder());
+  }));
 ```
 
-  - **As an output of another `BeamSql` query**. Details in the next section.
+  - **As an output of another `SqlTransform`**. Details in the next section.
 
-Once you have a `PCollection` in hand, you may use the `BeamSql` APIs to 
apply SQL queries to it.
+Once you have a `PCollection` in hand, you may use `SqlTransform` to 
apply SQL queries to it.
 
-## BeamSql transform
+## SqlTransform
 
-`BeamSql.query(queryString)` method is the only API to create a `PTransform`
+[`SqlTransform.query(queryString)`]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html) method is 
the only API to create a `PTransform`
 from a string representation of the SQL query. You can apply this `PTransform`
 to either a single `PCollection` or a `PCollectionTuple` which holds multiple
 `PCollections`:
@@ -122,7 +119,7 @@ to either a single `PCollection` or a `PCollectionTuple` 
which holds multiple
   - when applying to a single `PCollection` it can be referenced via the table 
name `PCOLLECTION` in the query:
 ```java
 PCollection filteredNames = testApps.apply(
-BeamSql.query(
+SqlTransform.query(
   "SELECT appId, description, rowtime "
 + "FROM PCOLLECTION "
 + "WHERE id=1"));
@@ -154,7 +151,7 @@ to either a single `PCollection` or a `PCollectionTuple` 
which holds multiple
 // and average rating per app 
 // by joining two PCollections
 PCollection output = namesAndFoods.apply(
-BeamSql.query(
+SqlTransform.query(
 "SELECT Names.appId, 

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133329
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:08
Start Date: 09/Aug/18 22:08
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#issuecomment-411913712
 
 
   java run precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133329)
Time Spent: 3h  (was: 2h 50m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 02/02: This closes #529

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7ff1a24a15be5e5f2b52473f3f5a9bce5c309f78
Merge: f5fde01 8b15877
Author: Mergebot 
AuthorDate: Thu Aug 9 22:08:57 2018 +

This closes #529

 src/documentation/dsls/sql/overview.md|  2 +-
 src/documentation/dsls/sql/walkthrough.md | 23 ++-
 2 files changed, 11 insertions(+), 14 deletions(-)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=10=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-10
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:08
Start Date: 09/Aug/18 22:08
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#issuecomment-411913712
 
 
   java run precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 10)
Time Spent: 3h 10m  (was: 3h)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133328
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 22:07
Start Date: 09/Aug/18 22:07
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#discussion_r209094143
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 ##
 @@ -1170,4 +1171,32 @@ public void testTimestampMinusInterval() throws 
Exception {
 parseDate("1983-01-19 01:01:58"));
 checker.buildRunAndCheck();
   }
+
+  @Test
+  @SqlOperatorTest(name = "CASE", kind = "CASE")
+  @SqlOperatorTest(name = "NULLIF", kind = "NULLIF")
+  @SqlOperatorTest(name = "COALESCE", kind = "COALESCE")
+  public void testConditionalOperatorsAndFunctions() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END", "hello")
+.addExpr(
+"CASE 2 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"world")
+.addExpr(
+"CASE 3 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"bond")
+.addExpr("CASE " + "WHEN 1 = 1 THEN 'hello' " + "ELSE 'world' 
END", "hello")
+.addExpr("CASE " + "WHEN 1 > 1 THEN 'hello' " + "ELSE 'world' 
END", "world")
+.addExpr("NULLIF(5, 4) ", 5)
+.addExpr("NULLIF(4, 5) ", 4)
+.addExpr("NULLIF(5, 5)", null, FieldType.INT32)
 
 Review comment:
   It won't work neither because of 
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java#L156
   
   
   So `null` cannot call `.getClass` because of NullPointerException even we 
convert it. (Compiler does not buy this conversion).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133328)
Time Spent: 2h 50m  (was: 2h 40m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133319
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:54
Start Date: 09/Aug/18 21:54
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6174: 
[BEAM-5106][SQL]test conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#discussion_r209091180
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 ##
 @@ -1170,4 +1171,32 @@ public void testTimestampMinusInterval() throws 
Exception {
 parseDate("1983-01-19 01:01:58"));
 checker.buildRunAndCheck();
   }
+
+  @Test
+  @SqlOperatorTest(name = "CASE", kind = "CASE")
+  @SqlOperatorTest(name = "NULLIF", kind = "NULLIF")
+  @SqlOperatorTest(name = "COALESCE", kind = "COALESCE")
+  public void testConditionalOperatorsAndFunctions() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END", "hello")
+.addExpr(
+"CASE 2 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"world")
+.addExpr(
+"CASE 3 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"bond")
+.addExpr("CASE " + "WHEN 1 = 1 THEN 'hello' " + "ELSE 'world' 
END", "hello")
+.addExpr("CASE " + "WHEN 1 > 1 THEN 'hello' " + "ELSE 'world' 
END", "world")
+.addExpr("NULLIF(5, 4) ", 5)
+.addExpr("NULLIF(4, 5) ", 4)
+.addExpr("NULLIF(5, 5)", null, FieldType.INT32)
 
 Review comment:
   I don't think I understand. If you mark the `expected` param of the 
`addExpr` as `@Nullable`, then you can have:
   
   ```
   new ExpressionChecker()
 .addExpr("expr", (Integer) null)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133319)
Time Spent: 2h 40m  (was: 2.5h)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5122:

Description: 
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]

Test flakes with timeout of getting update on pubsub:

java.lang.AssertionError: Did not receive signal on 
projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
 in 60s at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
 at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
 at 
org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]

 

  was:
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]

Test flakes with timeout of getting update on pubsub:

java.lang.AssertionError: Did not receive signal on 
projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
 in 60s at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
 at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
 at 
org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)

 


> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Major
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[Mikhail Gryzykhin (JIRA)]

Mikhail Gryzykhin created BEAM-5122:
---

 Summary: beam_PostCommit_Java_GradleBuild 
org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
 Key: BEAM-5122
 URL: https://issues.apache.org/jira/browse/BEAM-5122
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Mikhail Gryzykhin
Assignee: Anton Kedin


[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]

Test flakes with timeout of getting update on pubsub:

java.lang.AssertionError: Did not receive signal on 
projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
 in 60s at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
 at 
org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
 at 
org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133315
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:45
Start Date: 09/Aug/18 21:45
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#discussion_r209088713
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 ##
 @@ -152,12 +215,14 @@ private static ExpressionTestCase of(
 private transient List exps = new ArrayList<>();
 
 public ExpressionChecker addExpr(String expression, Object expectedValue) {
-  // Because of erasure, we can only automatically infer non-parameterized 
types
   TypeName resultTypeName = 
JAVA_CLASS_TO_TYPENAME.get(expectedValue.getClass());
   checkArgument(
   resultTypeName != null,
-  "Could not infer a Beam type for %s."
-  + " Parameterized types must be provided explicitly.");
+  String.format(
+  "The type of the expected object '%s' is unknown in 
'addExpr(String %s, Object %s)'"
 
 Review comment:
   this `addExpr` example can be confusing now, e.g. if you call this method 
with something like `addExpr("++b ", Arrays. toList(myObj1, ...)`, then 
the error message would be something like `addExpr(String ++b, Object 
[com.blah.myObj@af213])` which looks bad. I'd rather just leave the parameter 
names.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133315)
Time Spent: 4.5h  (was: 4h 20m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133316
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:45
Start Date: 09/Aug/18 21:45
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#discussion_r209087505
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 ##
 @@ -131,6 +192,8 @@ private static ExpressionTestCase of(
 abstract Object expectedResult();
 
 abstract FieldType resultFieldType();
+
+// abstract boolean isCompareNonExactNumber();
 
 Review comment:
   remove this


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133316)
Time Spent: 4h 40m  (was: 4.5h)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4658.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=133312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133312
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:44
Start Date: 09/Aug/18 21:44
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6187: [BEAM-4658] Follow 
up on PR comments from #6050
URL: https://github.com/apache/beam/pull/6187
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
index 3215691363c..2ddd80afae8 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
@@ -81,6 +81,7 @@
   private static final CompatibilityChecker DEFAULT_COMPATIBILITY_CHECKER =
   GreedyPCollectionFusers::unknownTransformCompatibility;
 
+  /** Returns true if the PTransform node for the given input PCollection can 
be fused across. */
   public static boolean canFuse(
   PTransformNode transformNode,
   Environment environment,
@@ -92,6 +93,10 @@ public static boolean canFuse(
 .canFuse(transformNode, environment, candidate, stagePCollections, 
pipeline);
   }
 
+  /**
+   * Returns true if the two PTransforms are compatible such that they can be 
executed in the same
+   * environment.
+   */
   public static boolean isCompatible(
   PTransformNode left, PTransformNode right, QueryablePipeline pipeline) {
 CompatibilityChecker leftChecker =
@@ -184,7 +189,8 @@ private static boolean canFuseParDo(
 
   private static boolean parDoCompatibility(
   PTransformNode parDo, PTransformNode other, QueryablePipeline pipeline) {
-// Implicitly true if we are attempting to fuse against oneself. This is 
for timer PCollection which create a loop.
+// Implicitly true if we are attempting to fuse against oneself. This case 
comes up for
+// PCollections representing timers since they create a self-loop in the 
graph.
 return parDo.equals(other)
 // This is a convenience rather than a strict requirement. In general, 
a ParDo that consumes
 // side inputs can be fused with other transforms in the same 
environment which are not
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/TimerReference.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/TimerReference.java
index 36dd3284a6a..0c6bf3a36bc 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/TimerReference.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/TimerReference.java
@@ -21,8 +21,9 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi;
 
 /**
- * A reference to a timer. This includes the PTransform that references the 
timer as well as the
- * PCollection referenced. Both are necessary in order to fully resolve a 
timer.
+ * Contains references to components relevant for runners during execution for 
timers. The
+ * referenced PTransform specifies the timer specification while the 
PCollection specifies the
+ * encoding representation.
  */
 @AutoValue
 public abstract class TimerReference {
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStageTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStageTest.java
index 4920e17cba9..41b1e6c1d4a 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStageTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStageTest.java
@@ -49,9 +49,9 @@ public void ofFullComponentsOnlyHasStagePTransforms() throws 
Exception {
 PTransform.newBuilder()
 .putInputs("input", "input.out")
 .putInputs("side_input", "sideInput.in")
-.putInputs("timer", "timer.out")
+.putInputs("timer", "timer.pc")
 .putOutputs("output", "output.out")
-.putOutputs("timer", "timer.out")
+.putOutputs("timer", 

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133311
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:43
Start Date: 09/Aug/18 21:43
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6176: [[BEAM-5092] Row 
comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#issuecomment-411907342
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133311)
Time Spent: 7h  (was: 6h 50m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133308
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:42
Start Date: 09/Aug/18 21:42
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209088235
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
 ##
 @@ -1394,5 +1420,31 @@ artifactId=${project.name}
 args argsNeeded
   }
 }
+
+
+/** 
***/
+
+// Method to create the PortableValidatesRunnerTask.
+// The method takes PortableValidatesRunnerConfiguration as parameter.
+project.ext.createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfiguration : new 
PortableValidatesRunnerConfiguration()
+  def name = config.name
+  def beamTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+  ]
+  if(config.jobServerConfig){
+
beamTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+  }
+  project.tasks.create(name: name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", 
JsonOutput.toJson(beamTestPipelineOptions)
+classpath = config.testClasspathConfiguration
+testClassesDirs = 
project.files(project.project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
 project.project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
 
 Review comment:
   OK, that sounds good for now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133308)
Time Spent: 12h  (was: 11h 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133295
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:39
Start Date: 09/Aug/18 21:39
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209087516
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
 ##
 @@ -1394,5 +1420,31 @@ artifactId=${project.name}
 args argsNeeded
   }
 }
+
+
+/** 
***/
+
+// Method to create the PortableValidatesRunnerTask.
+// The method takes PortableValidatesRunnerConfiguration as parameter.
+project.ext.createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfiguration : new 
PortableValidatesRunnerConfiguration()
+  def name = config.name
+  def beamTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+  ]
+  if(config.jobServerConfig){
+
beamTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+  }
+  project.tasks.create(name: name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", 
JsonOutput.toJson(beamTestPipelineOptions)
+classpath = config.testClasspathConfiguration
+testClassesDirs = 
project.files(project.project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
 project.project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
 
 Review comment:
   I suppose the gradle task will fail verbosely if the down stream project 
does not have dependency on :beam-sdks-java-core 
   The best I can think of at the moment is just document expectation for 
:beam-sdks-java-core dependency.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133295)
Time Spent: 11h 50m  (was: 11h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133294
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:38
Start Date: 09/Aug/18 21:38
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6175: [BEAM-5103][SQL]test 
aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-411906000
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133294)
Time Spent: 4h 20m  (was: 4h 10m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133293
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:37
Start Date: 09/Aug/18 21:37
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411905675
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133293)
Time Spent: 0.5h  (was: 20m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133288
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209082874
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/sideinput.go
 ##
 @@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package graphx
+
+import (
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+)
+
+// Side input support
+//
+// The Beam model supports only MultiMap side input, so for
+// iterable side input (= the only kind in Go so far) we must
+// implicitly use a fixed key (""). We use a special execution
+// unit as well to handle nested KVs.
+
+const (
+   URNIterableSideInputKey = "beam:go:transform:iterablesideinputkey:v1"
+)
+
+// MakeBytesKeyedCoder returns KV<[]byte,A,> for any coder,
+// even if the coder is already a KV coder.
+func MakeBytesKeyedCoder(c *coder.Coder) *coder.Coder {
 
 Review comment:
   Any reason this should be exported? Or otherwise be in this file alone, 
rather than in translate.go where it's used?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133288)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133290
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209075226
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/statemgr.go
 ##
 @@ -0,0 +1,310 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package harness
+
+import (
+   "context"
+   "fmt"
+   "io"
+   "sync"
+   "sync/atomic"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/proto"
+   "github.com/pkg/errors"
+)
+
+// ScopedStateManager scopes the global gRPC state manager to a single 
instruction.
 
 Review comment:
   Typo: ScopedSideInputReader,
   and the rest of the comment doesn't appear to be for the 
ScopedSideInputReader at all.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133290)
Time Spent: 1h 10m  (was: 1h)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133287
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209075357
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/statemgr.go
 ##
 @@ -0,0 +1,310 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package harness
+
+import (
+   "context"
+   "fmt"
+   "io"
+   "sync"
+   "sync/atomic"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/proto"
+   "github.com/pkg/errors"
+)
+
+// ScopedStateManager scopes the global gRPC state manager to a single 
instruction.
+// The indirection makes it easier to control access.
+type ScopedSideInputReader struct {
 
 Review comment:
   This is only used within this package. Does it need to be exported at all? 
Same question for most of the other Exported types in this file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133287)
Time Spent: 50m  (was: 40m)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133292
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209083489
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/sideinput.go
 ##
 @@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+   "context"
+   "fmt"
+   "io"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+// This file contains support for side input.
+
+const IterableSideInputKey = ""
+
+// SideInputAdapter provides a concrete ReStream from a low-level side input 
reader. It
+// encapsulates StreamID and coding as needed.
+type SideInputAdapter interface {
+   NewIterable(ctx context.Context, reader SideInputReader, w 
typex.Window) (ReStream, error)
+}
+
+type sideInputAdapter struct {
+   sid StreamID
+   wc  WindowEncoder
+   kc  ElementEncoder
+   ec  ElementDecoder
+}
+
+// NewSideInputAdapter returns a side input adapter for the given StreamID and 
coder.
+// It expects a W> coder, because the protocol supports MultiSet 
access only.
+func NewSideInputAdapter(sid StreamID, c *coder.Coder) SideInputAdapter {
+   if !coder.IsW(c) || !coder.IsKV(coder.SkipW(c)) {
+   panic(fmt.Sprintf("expected WKV coder for side input: %v", c))
+   }
+
+   wc := MakeWindowEncoder(c.Window)
+   kc := MakeElementEncoder(coder.SkipW(c).Components[0])
+   ec := MakeElementDecoder(coder.SkipW(c).Components[1])
+   return {sid: sid, wc: wc, kc: kc, ec: ec}
+}
+
+func (s *sideInputAdapter) NewIterable(ctx context.Context, reader 
SideInputReader, w typex.Window) (ReStream, error) {
+   key, err := EncodeElement(s.kc, []byte(IterableSideInputKey))
+   if err != nil {
+   return nil, err
+   }
+   win, err := EncodeWindow(s.wc, w)
+   if err != nil {
+   return nil, err
+   }
+   return {
+   open: func() (Stream, error) {
+   r, err := reader.Open(ctx, s.sid, key, win)
+   if err != nil {
+   return nil, err
+   }
+   return {r: r, ec: s.ec}, nil
+   },
+   }, nil
+}
+
+func (s *sideInputAdapter) String() string {
+   return fmt.Sprintf("SideInputAdapter[%v]", s.sid)
+}
+
+// proxyReStream is a simple wrapper of an open function.
+type proxyReStream struct {
+   open func() (Stream, error)
+}
+
+func (p *proxyReStream) Open() (Stream, error) {
+   return p.open()
+}
+
+// elementStream exposes a Stream from decoding elements.
+type elementStream struct {
+   r  io.ReadCloser
+   ec ElementDecoder
+}
+
+func (s *elementStream) Close() error {
+   return s.r.Close()
+}
+
+func (s *elementStream) Read() (FullValue, error) {
+   // We should see a stream of unwindowed values -- no sizes, no key.
+   return s.ec.Decode(s.r)
+}
+
+// FixedKey transform any value into KV for a fixed K.
+type FixedKey struct {
 
 Review comment:
   Consider not exporting this, unless necessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133292)
Time Spent: 1h 20m  (was: 1h 10m)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: 

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133291
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209082719
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/sideinput.go
 ##
 @@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package graphx
+
+import (
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+)
+
+// Side input support
+//
+// The Beam model supports only MultiMap side input, so for
+// iterable side input (= the only kind in Go so far) we must
+// implicitly use a fixed key (""). We use a special execution
+// unit as well to handle nested KVs.
+
+const (
+   URNIterableSideInputKey = "beam:go:transform:iterablesideinputkey:v1"
 
 Review comment:
   Consider putting this with the rest of the constants. It's odd that it's a) 
by itself b) not in the file where it's used.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133291)
Time Spent: 1h 10m  (was: 1h)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133289
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209079934
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/datamgr.go
 ##
 @@ -25,46 +25,78 @@ import (
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
"github.com/apache/beam/sdks/go/pkg/beam/log"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
-   "google.golang.org/grpc"
 )
 
 const (
chunkSize   = int(4e6) // Bytes to put in a single gRPC message. Max is 
slightly higher.
bufElements = 20   // Number of chunks buffered per reader.
 )
 
-// This is a reduced version of the full gRPC interface to help with testing.
-// TODO(wcn): need a compile-time assertion to make sure this stays synced 
with what's
-// in pb.BeamFnData_DataClient
-type dataClient interface {
-   Send(*pb.Elements) error
-   Recv() (*pb.Elements, error)
+// ScopedDataManager scopes the global gRPC data manager to a single 
instruction.
+// The indirection makes it easier to control access.
+type ScopedDataManager struct {
 
 Review comment:
   This is only used within the same package, consider not exporting it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133289)
Time Spent: 1h  (was: 50m)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3286) Go SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3286?focusedWorklogId=133286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133286
 ]

ASF GitHub Bot logged work on BEAM-3286:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:36
Start Date: 09/Aug/18 21:36
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6197: 
[BEAM-3286] Add preliminary Go support for side input
URL: https://github.com/apache/beam/pull/6197#discussion_r209072485
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/statemgr.go
 ##
 @@ -0,0 +1,310 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package harness
+
+import (
+   "context"
+   "fmt"
+   "io"
+   "sync"
+   "sync/atomic"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/proto"
+   "github.com/pkg/errors"
+)
+
+// ScopedStateManager scopes the global gRPC state manager to a single 
instruction.
+// The indirection makes it easier to control access.
+type ScopedSideInputReader struct {
+   mgr*StateChannelManager
+   instID string
+
+   opened []io.Closer // track open readers to force close all
+   closed bool
+   mu sync.Mutex
+}
+
+func NewScopedSideInputReader(mgr *StateChannelManager, instID string) 
*ScopedSideInputReader {
 
 Review comment:
   Exported functions, even relatively obvious ones, should probably have a 
comment
   // NewScopedSideInputReader ...
   
   Alternatively, this only seems to be invoked in this package. Does it need 
to be exported at all?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133286)
Time Spent: 40m  (was: 0.5h)

> Go SDK support for portable side input
> --
>
> Key: BEAM-3286
> URL: https://issues.apache.org/jira/browse/BEAM-3286
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5044) Project beam_PostCommit_Python_Verify: FnApiRunnerTestWithGrpc: BeamAssertException: Failed assert: ['a'] == ['a', 'b'] [while running 'assert_that/Match']

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5044:
---

Assignee: (was: Robert Bradshaw)

> Project beam_PostCommit_Python_Verify: FnApiRunnerTestWithGrpc: 
> BeamAssertException: Failed assert: ['a'] == ['a', 'b'] [while running 
> 'assert_that/Match']
> ---
>
> Key: BEAM-5044
> URL: https://issues.apache.org/jira/browse/BEAM-5044
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/5611/consoleFull
> test_assert_that 
> (apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithGrpc) 
> ... Traceback (most recent call last):
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
>  line 134, in _execute
>  response = task()
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
>  line 169, in 
>  self._execute(lambda: worker.do_instruction(work), work)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
>  line 215, in do_instruction
>  request.instruction_id)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
>  line 237, in process_bundle
>  processor.process_bundle(instruction_id)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/bundle_processor.py",
>  line 299, in process_bundle
>  input_op.process_encoded(data.data)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/bundle_processor.py",
>  line 120, in process_encoded
>  self.output(decoded_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/operations.py",
>  line 167, in output
>  cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/operations.py",
>  line 87, in receive
>  cython.cast(Operation, consumer).process(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/operations.py",
>  line 388, in process
>  self.dofn_receiver.receive(o)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 591, in receive
>  self.process(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 597, in process
>  self._reraise_augmented(exn)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 595, in process
>  self.do_fn_invoker.invoke_process(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 475, in invoke_process
>  windowed_value, additional_args, additional_kwargs, output_processor)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 525, in _invoke_per_window
>  windowed_value, self.process_method(*args_for_process))
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 700, in process_outputs
>  self.main_receivers.receive(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/operations.py",
>  line 87, in receive
>  cython.cast(Operation, consumer).process(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/worker/operations.py",
>  line 388, in process
>  self.dofn_receiver.receive(o)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 591, in receive
>  self.process(windowed_value)
>  File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/common.py",
>  line 597, in process
>  

[jira] [Updated] (BEAM-5116) beam_PostCommit_Java_GradleBuild :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed build. System failure.

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5116:

Priority: Minor  (was: Major)

> beam_PostCommit_Java_GradleBuild 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
> build. System failure.
> ---
>
> Key: BEAM-5116
> URL: https://issues.apache.org/jira/browse/BEAM-5116
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Minor
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1214/]
> Find log subset below. Suspect is fail is due to memory limitations or zip 
> source availability/integrity.
> > Task :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest 
> > FAILED
> Build cache key for task 
> ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' is 
> 12c45ad8abd6cf3734c37d0c2592d429
> Task ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' 
> is not up-to-date because:
>   No history is available.
> Starting process 'Gradle Test Executor 129'. Working directory: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java
>  Command: /usr/local/asfpackages/java/jdk1.8.0_172/bin/java 
> -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-end-to-end-tests"]
>  
> -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
>  -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=US 
> -Duser.language=en -Duser.variant -ea -cp 
> /home/jenkins/.gradle/caches/4.8/workerMain/gradle-worker.jar 
> worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
> Executor 129'
> Successfully started process 'Gradle Test Executor 129'
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7fc14674a4e2, pid=7054, tid=0x7fc0c1150700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_172-b11) (build 
> 1.8.0_172-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x124e2]  newEntry+0x62
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java/hs_err_pid7054.log
> Compiled method (nm)1382  211 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7fc1313495d0,0x7fc131349940] = 880
>  relocation [0x7fc1313496f8,0x7fc131349740] = 72
>  main code  [0x7fc131349740,0x7fc131349940] = 512
> #
> # If you would like to submit a bug report, please visit:
> #   
> [http://bugreport.java.com/bugreport/crash.jsp]
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5116) beam_PostCommit_Java_GradleBuild :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed build. System failure.

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575411#comment-16575411
 ] 

Mikhail Gryzykhin commented on BEAM-5116:
-

Synced with Luke if he had ideas out of head.

We can look into enabling core dump on build failure and go further from there.

I'm reducing priority on this issue, until it repeats itself.

> beam_PostCommit_Java_GradleBuild 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
> build. System failure.
> ---
>
> Key: BEAM-5116
> URL: https://issues.apache.org/jira/browse/BEAM-5116
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1214/]
> Find log subset below. Suspect is fail is due to memory limitations or zip 
> source availability/integrity.
> > Task :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest 
> > FAILED
> Build cache key for task 
> ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' is 
> 12c45ad8abd6cf3734c37d0c2592d429
> Task ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' 
> is not up-to-date because:
>   No history is available.
> Starting process 'Gradle Test Executor 129'. Working directory: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java
>  Command: /usr/local/asfpackages/java/jdk1.8.0_172/bin/java 
> -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-end-to-end-tests"]
>  
> -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
>  -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=US 
> -Duser.language=en -Duser.variant -ea -cp 
> /home/jenkins/.gradle/caches/4.8/workerMain/gradle-worker.jar 
> worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
> Executor 129'
> Successfully started process 'Gradle Test Executor 129'
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7fc14674a4e2, pid=7054, tid=0x7fc0c1150700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_172-b11) (build 
> 1.8.0_172-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x124e2]  newEntry+0x62
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java/hs_err_pid7054.log
> Compiled method (nm)1382  211 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7fc1313495d0,0x7fc131349940] = 880
>  relocation [0x7fc1313496f8,0x7fc131349740] = 72
>  main code  [0x7fc131349740,0x7fc131349940] = 512
> #
> # If you would like to submit a bug report, please visit:
> #   
> [http://bugreport.java.com/bugreport/crash.jsp]
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5116) beam_PostCommit_Java_GradleBuild :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed build. System failure.

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5116:

Summary: beam_PostCommit_Java_GradleBuild 
:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
build. System failure.  (was: beam_PostCommit_Java_GradleBuild 
:beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
build. Unclear reason.)

> beam_PostCommit_Java_GradleBuild 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
> build. System failure.
> ---
>
> Key: BEAM-5116
> URL: https://issues.apache.org/jira/browse/BEAM-5116
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1214/]
> Find log subset below. Suspect is fail is due to memory limitations or zip 
> source availability/integrity.
> > Task :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest 
> > FAILED
> Build cache key for task 
> ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' is 
> 12c45ad8abd6cf3734c37d0c2592d429
> Task ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' 
> is not up-to-date because:
>   No history is available.
> Starting process 'Gradle Test Executor 129'. Working directory: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java
>  Command: /usr/local/asfpackages/java/jdk1.8.0_172/bin/java 
> -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-end-to-end-tests"]
>  
> -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
>  -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=US 
> -Duser.language=en -Duser.variant -ea -cp 
> /home/jenkins/.gradle/caches/4.8/workerMain/gradle-worker.jar 
> worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
> Executor 129'
> Successfully started process 'Gradle Test Executor 129'
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7fc14674a4e2, pid=7054, tid=0x7fc0c1150700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_172-b11) (build 
> 1.8.0_172-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x124e2]  newEntry+0x62
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java/hs_err_pid7054.log
> Compiled method (nm)1382  211 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7fc1313495d0,0x7fc131349940] = 880
>  relocation [0x7fc1313496f8,0x7fc131349740] = 72
>  main code  [0x7fc131349740,0x7fc131349940] = 512
> #
> # If you would like to submit a bug report, please visit:
> #   
> [http://bugreport.java.com/bugreport/crash.jsp]
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5116) beam_PostCommit_Java_GradleBuild :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed build. Unclear reason.

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5116:
---

Assignee: (was: Luke Cwik)

> beam_PostCommit_Java_GradleBuild 
> :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest Failed 
> build. Unclear reason.
> ---
>
> Key: BEAM-5116
> URL: https://issues.apache.org/jira/browse/BEAM-5116
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1214/]
> Find log subset below. Suspect is fail is due to memory limitations or zip 
> source availability/integrity.
> > Task :beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest 
> > FAILED
> Build cache key for task 
> ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' is 
> 12c45ad8abd6cf3734c37d0c2592d429
> Task ':beam-runners-google-cloud-dataflow-java:examplesJavaIntegrationTest' 
> is not up-to-date because:
>   No history is available.
> Starting process 'Gradle Test Executor 129'. Working directory: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java
>  Command: /usr/local/asfpackages/java/jdk1.8.0_172/bin/java 
> -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-end-to-end-tests"]
>  
> -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
>  -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=US 
> -Duser.language=en -Duser.variant -ea -cp 
> /home/jenkins/.gradle/caches/4.8/workerMain/gradle-worker.jar 
> worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
> Executor 129'
> Successfully started process 'Gradle Test Executor 129'
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7fc14674a4e2, pid=7054, tid=0x7fc0c1150700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_172-b11) (build 
> 1.8.0_172-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x124e2]  newEntry+0x62
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/runners/google-cloud-dataflow-java/hs_err_pid7054.log
> Compiled method (nm)1382  211 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7fc1313495d0,0x7fc131349940] = 880
>  relocation [0x7fc1313496f8,0x7fc131349740] = 72
>  main code  [0x7fc131349740,0x7fc131349940] = 512
> #
> # If you would like to submit a bug report, please visit:
> #   
> [http://bugreport.java.com/bugreport/crash.jsp]
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4699) BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-4699.

   Resolution: Duplicate
Fix Version/s: 2.7.0

The issue is a duplicate of BEAM-4810
Please re open if you see the issue again.

 

> BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake
> 
>
> Key: BEAM-4699
> URL: https://issues.apache.org/jira/browse/BEAM-4699
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
> Fix For: 2.7.0
>
>
> I've seen a few transient failures from 
> {{BeamFileSystemArtifactServicesTest}}. I don't recall if they are all 
> {{putArtifactsSingleSmallFileTest}} or how often they occur.
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/
> {code}
> java.io.FileNotFoundException: 
> /tmp/junit8499382858780569091/staging/123/artifacts/artifact_c147efcfc2d7ea666a9e4f5187b115c90903f0fc896a56df9a6ef5d8f3fc9f31
>  (No such file or directory)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5110) Reconile Flink JVM singleton management with deployment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5110?focusedWorklogId=133281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133281
 ]

ASF GitHub Bot logged work on BEAM-5110:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:23
Start Date: 09/Aug/18 21:23
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #6189: [BEAM-5110] 
Explicitly count the references for BatchFlinkExecutableStageContext …
URL: https://github.com/apache/beam/pull/6189#issuecomment-411902154
 
 
   @tweise It's both to reduce initialization time and share resources. The 
main resource being an SDK harness (Environment). Depending on the 
ContextFactory implementation, we may use a DockerEnvironmentFactory or 
something else (e.g., a KubernetesEnvironmentFactory down the line). I believe 
this is documented elsewhere in the design, but we can add this to the 
context's javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133281)
Time Spent: 1h 40m  (was: 1.5h)

> Reconile Flink JVM singleton management with deployment
> ---
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> [~angoenka] noticed through debugging that multiple instances of 
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when 
> executing in standalone cluster mode. This context factory is responsible for 
> maintaining singleton state across a TaskManager (JVM) in order to share SDK 
> Environments across workers in a given job. The multiple-loading breaks 
> singleton semantics and results in an indeterminate number of Environments 
> being created.
> It turns out that the [Flink classloading 
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
>  is determined by deployment mode. Note that "user code" as referenced by 
> this link is actually the Flink job server jar. Actual end-user code lives 
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using 
> file locks and/or additional gRPC servers), we need to force non-dynamic 
> classloading. For example, this happens when jobs are submitted to YARN for 
> one-off deployments via `flink run`. However, connecting to an existing 
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to 
> enforce) deployment modes that are consistent with our requirements, or (if 
> possible) create a custom classloader that enforces singleton loading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5110) Reconile Flink JVM singleton management with deployment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5110?focusedWorklogId=133275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133275
 ]

ASF GitHub Bot logged work on BEAM-5110:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:13
Start Date: 09/Aug/18 21:13
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6189: [BEAM-5110] Explicitly 
count the references for BatchFlinkExecutableStageContext …
URL: https://github.com/apache/beam/pull/6189#issuecomment-411899136
 
 
   @bsidhom I meanwhile realized that also while working on streaming side 
inputs. But it isn't clear to me why the context needs to be shared. Is that 
intended to be an optimization for batch (to reduce the initialization time)? 
If so, then please document this? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133275)
Time Spent: 1.5h  (was: 1h 20m)

> Reconile Flink JVM singleton management with deployment
> ---
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [~angoenka] noticed through debugging that multiple instances of 
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when 
> executing in standalone cluster mode. This context factory is responsible for 
> maintaining singleton state across a TaskManager (JVM) in order to share SDK 
> Environments across workers in a given job. The multiple-loading breaks 
> singleton semantics and results in an indeterminate number of Environments 
> being created.
> It turns out that the [Flink classloading 
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
>  is determined by deployment mode. Note that "user code" as referenced by 
> this link is actually the Flink job server jar. Actual end-user code lives 
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using 
> file locks and/or additional gRPC servers), we need to force non-dynamic 
> classloading. For example, this happens when jobs are submitted to YARN for 
> one-off deployments via `flink run`. However, connecting to an existing 
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to 
> enforce) deployment modes that are consistent with our requirements, or (if 
> possible) create a custom classloader that enforces singleton loading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133274
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:11
Start Date: 09/Aug/18 21:11
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411898621
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133274)
Time Spent: 20m  (was: 10m)

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Closed] (BEAM-4807) Upgrade calcite to 1.17.0

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-4807.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Upgrade calcite to 1.17.0
> -
>
> Key: BEAM-4807
> URL: https://issues.apache.org/jira/browse/BEAM-4807
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We should upgrade calcite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4846) updateOfflineRepositoryRoot broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?focusedWorklogId=133255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133255
 ]

ASF GitHub Bot logged work on BEAM-4846:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:06
Start Date: 09/Aug/18 21:06
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6196: [BEAM-4846] Fixing 
updateOfflineRepository task. 
URL: https://github.com/apache/beam/pull/6196#issuecomment-411897059
 
 
   Run Java PreCommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133255)
Time Spent: 10m
Remaining Estimate: 0h

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> 

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133246
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411558103
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133246)
Time Spent: 1h 50m  (was: 1h 40m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133249
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411593759
 
 
   Run Python Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133249)
Time Spent: 2h 20m  (was: 2h 10m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133248
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411593017
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133248)
Time Spent: 2h 10m  (was: 2h)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133247
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411561045
 
 
   Run Python Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133247)
Time Spent: 2h  (was: 1h 50m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133251
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411845364
 
 
   Run Python Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133251)
Time Spent: 2h 40m  (was: 2.5h)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133252
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411849301
 
 
   Run Python Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133252)
Time Spent: 2h 50m  (was: 2h 40m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5109) Build nightly snapshot for Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5109?focusedWorklogId=133250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133250
 ]

ASF GitHub Bot logged work on BEAM-5109:


Author: ASF GitHub Bot
Created on: 09/Aug/18 21:03
Start Date: 09/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: yifanzou removed a comment on issue #6185: [BEAM-5109] 
create python nightly snapshots
URL: https://github.com/apache/beam/pull/6185#issuecomment-411843043
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133250)
Time Spent: 2.5h  (was: 2h 20m)

> Build nightly snapshot for Python SDK
> -
>
> Key: BEAM-5109
> URL: https://issues.apache.org/jira/browse/BEAM-5109
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5110) Reconile Flink JVM singleton management with deployment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5110?focusedWorklogId=133244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133244
 ]

ASF GitHub Bot logged work on BEAM-5110:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:52
Start Date: 09/Aug/18 20:52
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #6189: [BEAM-5110] 
Explicitly count the references for BatchFlinkExecutableStageContext …
URL: https://github.com/apache/beam/pull/6189#issuecomment-411893275
 
 
   @tweise We also need the factory to be pluggable for testing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133244)
Time Spent: 1h 20m  (was: 1h 10m)

> Reconile Flink JVM singleton management with deployment
> ---
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> [~angoenka] noticed through debugging that multiple instances of 
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when 
> executing in standalone cluster mode. This context factory is responsible for 
> maintaining singleton state across a TaskManager (JVM) in order to share SDK 
> Environments across workers in a given job. The multiple-loading breaks 
> singleton semantics and results in an indeterminate number of Environments 
> being created.
> It turns out that the [Flink classloading 
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
>  is determined by deployment mode. Note that "user code" as referenced by 
> this link is actually the Flink job server jar. Actual end-user code lives 
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using 
> file locks and/or additional gRPC servers), we need to force non-dynamic 
> classloading. For example, this happens when jobs are submitted to YARN for 
> one-off deployments via `flink run`. However, connecting to an existing 
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to 
> enforce) deployment modes that are consistent with our requirements, or (if 
> possible) create a custom classloader that enforces singleton loading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5073) Enable SortRemoveRule

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5073:
---
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-5119

> Enable SortRemoveRule
> -
>
> Key: BEAM-5073
> URL: https://issues.apache.org/jira/browse/BEAM-5073
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Enable SortRemoveRule by using RelCollationTraitDef.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5121) Investigate flattening issue of nested Row

[Rui Wang (JIRA)]

Rui Wang created BEAM-5121:
--

 Summary: Investigate flattening issue of nested Row
 Key: BEAM-5121
 URL: https://issues.apache.org/jira/browse/BEAM-5121
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4196) [SQL] Support Complex Types in DDL

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4196:
---
Issue Type: Sub-task  (was: New Feature)
Parent: BEAM-5120

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4710) Read complex types of data by SQL

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4710:
---
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-5120

> Read complex types of data by SQL
> -
>
> Key: BEAM-4710
> URL: https://issues.apache.org/jira/browse/BEAM-4710
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Support reading complex types of data by SQL. Typical complex types include 
> nested ROW, nested ARRAY, etc. 
>  
> Complex types might be different for different data sources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5120) Support Complex type in BeamSQL

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5120:
---
Description: 
We want have a smooth experience of complex type in BeamSQL.

 

For example, BeamSQL could support nested row in arbitrary levels 
(Row>>) in both read and write from/to arbitrary sources. 

  was:
We want have a smooth experience of complex type in BeamSQL.

 

For example, BeamSQL could support nested row in arbitrary levels 
(Row>>) and so on. 


> Support Complex type in BeamSQL
> ---
>
> Key: BEAM-5120
> URL: https://issues.apache.org/jira/browse/BEAM-5120
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> We want have a smooth experience of complex type in BeamSQL.
>  
> For example, BeamSQL could support nested row in arbitrary levels 
> (Row>>) in both read and write from/to arbitrary sources. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133234
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:39
Start Date: 09/Aug/18 20:39
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6176: [[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209070028
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
 ##
 @@ -31,8 +31,12 @@
  * This provider finds (recursively) all public getters and setters in a 
Java object, and creates
  * schemas and rows that bind to those fields. The field order in the schema 
is not guaranteed to
  * match the method order in the class. The Java object is expected to have 
implemented a correct
- * .equals() method. TODO: Validate equals() method is provided, and if not 
generate a "slow" equals
- * method based on the schema.
+ * .equals() and .hashCode method.s The equals method must be completely 
determined by the schema
 
 Review comment:
   Nit:
   
   "implemented a correct .equals() and .hashCode method.s" to
   "implemented correct equals() and hashCode() methods."
   
   (appears in other places in this commit too)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133234)
Time Spent: 6h 50m  (was: 6h 40m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5120) Support Complex type in BeamSQL

[Rui Wang (JIRA)]

Rui Wang created BEAM-5120:
--

 Summary: Support Complex type in BeamSQL
 Key: BEAM-5120
 URL: https://issues.apache.org/jira/browse/BEAM-5120
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang


We want have a smooth experience of complex type in BeamSQL.

 

For example, BeamSQL could support nested row in arbitrary levels 
(Row>>) and so on. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133232
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:37
Start Date: 09/Aug/18 20:37
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #6176: [[BEAM-5092] 
Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#issuecomment-41164
 
 
   Thanks @reuvenlax 
   
   A typo in the jdoc you just updated but otherwise LGTM (contributor not 
committer) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133232)
Time Spent: 6h 40m  (was: 6.5h)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133230
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:35
Start Date: 09/Aug/18 20:35
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6176: [[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209070028
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
 ##
 @@ -31,8 +31,12 @@
  * This provider finds (recursively) all public getters and setters in a 
Java object, and creates
  * schemas and rows that bind to those fields. The field order in the schema 
is not guaranteed to
  * match the method order in the class. The Java object is expected to have 
implemented a correct
- * .equals() method. TODO: Validate equals() method is provided, and if not 
generate a "slow" equals
- * method based on the schema.
+ * .equals() and .hashCode method.s The equals method must be completely 
determined by the schema
 
 Review comment:
   Nit:
   
   "method's." not "method.s"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133230)
Time Spent: 6h 20m  (was: 6h 10m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133231
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:35
Start Date: 09/Aug/18 20:35
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6176: [[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209070028
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
 ##
 @@ -31,8 +31,12 @@
  * This provider finds (recursively) all public getters and setters in a 
Java object, and creates
  * schemas and rows that bind to those fields. The field order in the schema 
is not guaranteed to
  * match the method order in the class. The Java object is expected to have 
implemented a correct
- * .equals() method. TODO: Validate equals() method is provided, and if not 
generate a "slow" equals
- * method based on the schema.
+ * .equals() and .hashCode method.s The equals method must be completely 
determined by the schema
 
 Review comment:
   Nit:
   
   "method's." not "method.s"
   
   (appears in other places in this commit too)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133231)
Time Spent: 6.5h  (was: 6h 20m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5110) Reconile Flink JVM singleton management with deployment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5110?focusedWorklogId=133229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133229
 ]

ASF GitHub Bot logged work on BEAM-5110:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:35
Start Date: 09/Aug/18 20:35
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6189: [BEAM-5110] 
Explicitly count the references for BatchFlinkExecutableStageContext …
URL: https://github.com/apache/beam/pull/6189#issuecomment-411888310
 
 
   We need to share the context mostly to keep track of SDKHarness process.
   Also the ExecutableStage starts control, logging, data, state services.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133229)
Time Spent: 1h 10m  (was: 1h)

> Reconile Flink JVM singleton management with deployment
> ---
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> [~angoenka] noticed through debugging that multiple instances of 
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when 
> executing in standalone cluster mode. This context factory is responsible for 
> maintaining singleton state across a TaskManager (JVM) in order to share SDK 
> Environments across workers in a given job. The multiple-loading breaks 
> singleton semantics and results in an indeterminate number of Environments 
> being created.
> It turns out that the [Flink classloading 
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
>  is determined by deployment mode. Note that "user code" as referenced by 
> this link is actually the Flink job server jar. Actual end-user code lives 
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using 
> file locks and/or additional gRPC servers), we need to force non-dynamic 
> classloading. For example, this happens when jobs are submitted to YARN for 
> one-off deployments via `flink run`. However, connecting to an existing 
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to 
> enforce) deployment modes that are consistent with our requirements, or (if 
> possible) create a custom classloader that enforces singleton loading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5119) Enable Calcite optimization rules

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang reassigned BEAM-5119:
--

Assignee: Rui Wang  (was: Xu Mingmin)

> Enable Calcite optimization rules
> -
>
> Key: BEAM-5119
> URL: https://issues.apache.org/jira/browse/BEAM-5119
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133228
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:32
Start Date: 09/Aug/18 20:32
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6176: 
[[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209069047
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -123,4 +124,27 @@ public int getFieldCount() {
   public Object getGetterTarget() {
 return getterTarget;
   }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null) {
+  return false;
+}
+if (o instanceof RowWithGetters) {
+  RowWithGetters other = (RowWithGetters) o;
+  return Objects.equals(getSchema(), other.getSchema())
+  && Objects.equals(getterTarget, other.getterTarget);
 
 Review comment:
   Also added to SchemaRegistry.registerPOJO/registerJavaBean


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133228)
Time Spent: 6h 10m  (was: 6h)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133223
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:24
Start Date: 09/Aug/18 20:24
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6176: 
[[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209066947
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -123,4 +124,27 @@ public int getFieldCount() {
   public Object getGetterTarget() {
 return getterTarget;
   }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null) {
+  return false;
+}
+if (o instanceof RowWithGetters) {
+  RowWithGetters other = (RowWithGetters) o;
+  return Objects.equals(getSchema(), other.getSchema())
+  && Objects.equals(getterTarget, other.getterTarget);
 
 Review comment:
   in JavaFieldSchema.java and JavaBeanSchema.java (as right now these are the 
classes that Beam users use when annotating Java classes for inference)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133223)
Time Spent: 6h  (was: 5h 50m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133221
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 09/Aug/18 20:18
Start Date: 09/Aug/18 20:18
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6176: [[BEAM-5092] Row comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#discussion_r209064859
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -123,4 +124,27 @@ public int getFieldCount() {
   public Object getGetterTarget() {
 return getterTarget;
   }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null) {
+  return false;
+}
+if (o instanceof RowWithGetters) {
+  RowWithGetters other = (RowWithGetters) o;
+  return Objects.equals(getSchema(), other.getSchema())
+  && Objects.equals(getterTarget, other.getterTarget);
 
 Review comment:
   Where is that please?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133221)
Time Spent: 5h 50m  (was: 5h 40m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4699) BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-4699:

Issue Type: Bug  (was: New Feature)

> BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake
> 
>
> Key: BEAM-4699
> URL: https://issues.apache.org/jira/browse/BEAM-4699
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>
> I've seen a few transient failures from 
> {{BeamFileSystemArtifactServicesTest}}. I don't recall if they are all 
> {{putArtifactsSingleSmallFileTest}} or how often they occur.
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/
> {code}
> java.io.FileNotFoundException: 
> /tmp/junit8499382858780569091/staging/123/artifacts/artifact_c147efcfc2d7ea666a9e4f5187b115c90903f0fc896a56df9a6ef5d8f3fc9f31
>  (No such file or directory)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4699) BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-4699:
---

Assignee: Ankur Goenka  (was: Henning Rohde)

> BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake
> 
>
> Key: BEAM-4699
> URL: https://issues.apache.org/jira/browse/BEAM-4699
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>
> I've seen a few transient failures from 
> {{BeamFileSystemArtifactServicesTest}}. I don't recall if they are all 
> {{putArtifactsSingleSmallFileTest}} or how often they occur.
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/
> {code}
> java.io.FileNotFoundException: 
> /tmp/junit8499382858780569091/staging/123/artifacts/artifact_c147efcfc2d7ea666a9e4f5187b115c90903f0fc896a56df9a6ef5d8f3fc9f31
>  (No such file or directory)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4636) Make beam.Run() (and/or friends) thread-safe.

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-4636.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Make beam.Run() (and/or friends) thread-safe.
> -
>
> Key: BEAM-4636
> URL: https://issues.apache.org/jira/browse/BEAM-4636
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Eduardo Morales
>Assignee: Henning Rohde
>Priority: Minor
> Fix For: 2.7.0
>
>
> It would be nice to be able to launch dataflow pipelines in parallel. 
> For example, here is my use case:
>  * I consume data produced by my clients/customers.
>  * I need to launch a pipeline on an event dispatch.
>  * I may receive multiple events, from multiple customers at the same time.
>  * Go code could be simpler if synchronization/cooperation wouldn't be needed 
> from goroutines handling each customer. In particular, setting options 
> through flags is cumbersome.
>  * Launching dataflow pipelines serially may not scale if I am able to sign 
> up many customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4699) BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-4699:

Labels: portability  (was: )

> BeamFileSystemArtifactServicesTest.putArtifactsSingleSmallFileTest flake
> 
>
> Key: BEAM-4699
> URL: https://issues.apache.org/jira/browse/BEAM-4699
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>
> I've seen a few transient failures from 
> {{BeamFileSystemArtifactServicesTest}}. I don't recall if they are all 
> {{putArtifactsSingleSmallFileTest}} or how often they occur.
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/
> {code}
> java.io.FileNotFoundException: 
> /tmp/junit8499382858780569091/staging/123/artifacts/artifact_c147efcfc2d7ea666a9e4f5187b115c90903f0fc896a56df9a6ef5d8f3fc9f31
>  (No such file or directory)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4636) Make beam.Run() (and/or friends) thread-safe.

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575337#comment-16575337
 ] 

Henning Rohde commented on BEAM-4636:
-

[~exm] I think this should be fixed. I also refactored the Go Dataflow runner 
into a dataflowlib and a driver (similarly to the universal runner), so that 
you can use the lib directly without flags.

> Make beam.Run() (and/or friends) thread-safe.
> -
>
> Key: BEAM-4636
> URL: https://issues.apache.org/jira/browse/BEAM-4636
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Eduardo Morales
>Assignee: Henning Rohde
>Priority: Minor
>
> It would be nice to be able to launch dataflow pipelines in parallel. 
> For example, here is my use case:
>  * I consume data produced by my clients/customers.
>  * I need to launch a pipeline on an event dispatch.
>  * I may receive multiple events, from multiple customers at the same time.
>  * Go code could be simpler if synchronization/cooperation wouldn't be needed 
> from goroutines handling each customer. In particular, setting options 
> through flags is cumbersome.
>  * Launching dataflow pipelines serially may not scale if I am able to sign 
> up many customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133209
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 19:56
Start Date: 09/Aug/18 19:56
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-411877648
 
 
   @akedin addressed your comments.  PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133209)
Time Spent: 3h 50m  (was: 3h 40m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133211
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 19:56
Start Date: 09/Aug/18 19:56
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-411877683
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133211)
Time Spent: 4h  (was: 3h 50m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133212
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 09/Aug/18 19:56
Start Date: 09/Aug/18 19:56
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-411877683
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133212)
Time Spent: 4h 10m  (was: 4h)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)