[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=131730&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131730
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 07/Aug/18 06:48
Start Date: 07/Aug/18 06:48
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6160: [BEAM-5092] Make 
sure that SchemaCoder overrides consistentWithEquals.
URL: https://github.com/apache/beam/pull/6160#issuecomment-410952147
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131730)
Time Spent: 40m  (was: 0.5h)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1192

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Java_Cron #196

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping bigdataoss to more recent version

--
[...truncated 12.82 MB...]
INFO: SQL:
SELECT CARDINALITY(`PCOLLECTION`.`f_stringArr`)
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[CARDINALITY($1)])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[CARDINALITY($t1)], EXPR$0=[$t2])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectRowsFromArrayOfRows STANDARD_ERROR
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_arrayOfRows=[$1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], f_arrayOfRows=[$t1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestLiteral 
STANDARD_ERROR
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `EXPR$0`.`EXPR$0`
FROM UNNEST(ARRAY['a', 'b', 'c']) AS `EXPR$0`
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[$0])
  Uncollect
LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
  LogicalValues(tuples=[[{ 0 }]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], EXPR$0=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestNamedLiteral 
STANDARD_ERROR
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `t`.`f_string`
FROM UNNEST(ARRAY['a', 'b', 'c']) AS `t` (`f_string`)
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_string=[$0])
  Uncollect
LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
  LogicalValues(tuples=[[{ 0 }]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], f_string=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectSingleRowFromArrayOfRows STANDARD_ERROR
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`[1]
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0$0=[ITEM($1, 1).f_rowString], EXPR$0$1=[ITEM($1, 
1).f_rowInt])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[1], expr#3=[ITEM($t1, $t2)], 
expr#4=[$t3.f_rowString], expr#5=[$t3.f_rowInt], EXPR$0$0=[$t4], EXPR$0$1=[$t5])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testProjectArrayField 
STANDARD_ERROR
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_int`, `PCOLLECTION`.`f_stringArr`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_int=[$0], f_stringArr=[$1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 07, 2018 6:11:38 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeam

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=131691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131691
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 07/Aug/18 02:15
Start Date: 07/Aug/18 02:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6160: [BEAM-5092] Make 
sure that SchemaCoder overrides consistentWithEquals.
URL: https://github.com/apache/beam/pull/6160#issuecomment-410910615
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131691)
Time Spent: 0.5h  (was: 20m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1191

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping bigdataoss to more recent version

--
[...truncated 19.96 MB...]
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:45.971Z: Checking required Cloud APIs are enabled.
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:46.280Z: Checking permissions granted to controller 
Service Account.
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:50.406Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:50.898Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.166Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.214Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.489Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.529Z: Elided trivial flatten 
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.581Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.627Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.675Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.722Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.764Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.810Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.854Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Aug 07, 2018 1:38:02 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-07T01:37:51.902Z: 

[jira] [Work logged] (BEAM-4809) Java preCommit and postCommit should build javadoc to check it builds ok

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4809?focusedWorklogId=131687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131687
 ]

ASF GitHub Bot logged work on BEAM-4809:


Author: ASF GitHub Bot
Created on: 07/Aug/18 01:31
Start Date: 07/Aug/18 01:31
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5970: [BEAM-4809] Build 
javadoc on :javaPreCommit and :javaPostCommit
URL: https://github.com/apache/beam/pull/5970#issuecomment-410903602
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131687)
Time Spent: 3h 40m  (was: 3.5h)

> Java preCommit and postCommit should build javadoc to check it builds ok
> 
>
> Key: BEAM-4809
> URL: https://issues.apache.org/jira/browse/BEAM-4809
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Java preCommit and postCommit should build javadoc to check it builds ok



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-5034] Bumping bigdataoss to more recent version

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7d0c99b8342009342318c5d729b56ec45d76752a
Merge: abb3d34 5613724
Author: Lukasz Cwik 
AuthorDate: Mon Aug 6 17:57:56 2018 -0700

[BEAM-5034] Bumping bigdataoss to more recent version

 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (abb3d34 -> 7d0c99b)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from abb3d34  [BEAM-1009] Upgrade mockito-core to 1.10.19
 add 5613724  Bumping bigdataoss to more recent version
 new 7d0c99b  [BEAM-5034] Bumping bigdataoss to more recent version

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5034) Bumping com.google.cloud.bigdataoss:gcsio and :util to 1.9.2 causes tests to fail

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5034?focusedWorklogId=131685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131685
 ]

ASF GitHub Bot logged work on BEAM-5034:


Author: ASF GitHub Bot
Created on: 07/Aug/18 00:57
Start Date: 07/Aug/18 00:57
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6147: [BEAM-5034] Bumping 
bigdataoss to more recent version
URL: https://github.com/apache/beam/pull/6147
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 05c0636b240..ee270a8aa94 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -297,7 +297,7 @@ class BeamModulePlugin implements Plugin {
 // Maven artifacts.
 def generated_grpc_beta_version = "0.19.0"
 def generated_grpc_ga_version = "1.18.0"
-def google_cloud_bigdataoss_version = "1.4.5"
+def google_cloud_bigdataoss_version = "1.9.0"
 def bigtable_version = "1.4.0"
 def google_clients_version = "1.23.0"
 def google_auth_version = "0.10.0"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131685)
Time Spent: 10m
Remaining Estimate: 0h

> Bumping com.google.cloud.bigdataoss:gcsio and :util to 1.9.2 causes tests to 
> fail
> -
>
> Key: BEAM-5034
> URL: https://issues.apache.org/jira/browse/BEAM-5034
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Garrett Jones
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Current version in Beam: 1.4.5
> Latest released version: 1.9.2
> Test failure:
> ```
> java.lang.NullPointerException at 
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.createRequest(GoogleCloudStorageReadChannel.java:966)
>  at 
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.getMetadata(GoogleCloudStorageReadChannel.java:765)
>  at 
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.initEncodingAndSize(GoogleCloudStorageReadChannel.java:754)
>  at 
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.(GoogleCloudStorageReadChannel.java:210)
>  at 
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.(GoogleCloudStorageReadChannel.java:172)
>  at 
> org.apache.beam.sdk.util.GcsUtilTest.testGCSChannelCloseIdempotent(GcsUtilTest.java:774)
> ```
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1190

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570954#comment-16570954
 ] 

Luke Cwik commented on BEAM-5035:
-

I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same failure by using [inotifywatch|https://linux.die.net/man/1/inotifywatch] 
(assuming you can reproduce this failure locally).

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ahmet Altay
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570954#comment-16570954
 ] 

Luke Cwik edited comment on BEAM-5035 at 8/7/18 12:15 AM:
--

I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same class of failure by using 
[inotifywatch|https://linux.die.net/man/1/inotifywatch] (assuming you can 
reproduce this failure locally).


was (Author: lcwik):
I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same failure by using [inotifywatch|https://linux.die.net/man/1/inotifywatch] 
(assuming you can reproduce this failure locally).

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ahmet Altay
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4826) Flink runner sends bad flatten to SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4826?focusedWorklogId=131678&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131678
 ]

ASF GitHub Bot logged work on BEAM-4826:


Author: ASF GitHub Bot
Created on: 07/Aug/18 00:13
Start Date: 07/Aug/18 00:13
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6132: [BEAM-4826] Sanitize 
pCollections before sending to SDKHarness
URL: https://github.com/apache/beam/pull/6132#issuecomment-410892140
 
 
   Makes sense.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131678)
Time Spent: 50m  (was: 40m)

> Flink runner sends bad flatten to SDK
> -
>
> Key: BEAM-4826
> URL: https://issues.apache.org/jira/browse/BEAM-4826
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> For a Go flatten test w/ 3 input, the Flink runner splits this into 3 bundle 
> descriptors. But it sends the original 3-input flatten but w/ 1 actual input 
> present in each bundle descriptor. This is inconsistent and the SDK shouldn't 
> expect dangling PCollections. In contrast, Dataflow removes the flatten when 
> it does the same split.
> Snippet:
> register: <
>   process_bundle_descriptor: <
> id: "3"
> transforms: <
>   key: "e4"
>   value: <
> unique_name: "github.com/apache/beam/sdks/go/pkg/beam.createFn'1"
> spec: <
>   urn: "urn:beam:transform:pardo:v1"
>   payload: [...]
> >
> inputs: <
>   key: "i0"
>   value: "n3"
> >
> outputs: <
>   key: "i0"
>   value: "n4"
> >
>   >
> >
> transforms: <
>   key: "e7"
>   value: <
> unique_name: "Flatten"
> spec: <
>   urn: "beam:transform:flatten:v1"
> >
> inputs: <
>   key: "i0"
>   value: "n2"
> >
> inputs: <
>   key: "i1"
>   value: "n4" . // <--- only one present.
> >
> inputs: <
>   key: "i2"
>   value: "n6"
> >
> outputs: <
>   key: "i0"
>   value: "n7"
> >
>   >
> >
> [...]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4826) Flink runner sends bad flatten to SDK

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570945#comment-16570945
 ] 

Luke Cwik commented on BEAM-4826:
-

This seems like a case where the GreedyPipelineFuser is incorrectly creating an 
ExecutableStage with a malformed flatten. Since the ExecutableStage contains 
the pipeline components, the flatten that is inserted into the ExecutableStage 
should have its inputs pruned there.

> Flink runner sends bad flatten to SDK
> -
>
> Key: BEAM-4826
> URL: https://issues.apache.org/jira/browse/BEAM-4826
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For a Go flatten test w/ 3 input, the Flink runner splits this into 3 bundle 
> descriptors. But it sends the original 3-input flatten but w/ 1 actual input 
> present in each bundle descriptor. This is inconsistent and the SDK shouldn't 
> expect dangling PCollections. In contrast, Dataflow removes the flatten when 
> it does the same split.
> Snippet:
> register: <
>   process_bundle_descriptor: <
> id: "3"
> transforms: <
>   key: "e4"
>   value: <
> unique_name: "github.com/apache/beam/sdks/go/pkg/beam.createFn'1"
> spec: <
>   urn: "urn:beam:transform:pardo:v1"
>   payload: [...]
> >
> inputs: <
>   key: "i0"
>   value: "n3"
> >
> outputs: <
>   key: "i0"
>   value: "n4"
> >
>   >
> >
> transforms: <
>   key: "e7"
>   value: <
> unique_name: "Flatten"
> spec: <
>   urn: "beam:transform:flatten:v1"
> >
> inputs: <
>   key: "i0"
>   value: "n2"
> >
> inputs: <
>   key: "i1"
>   value: "n4" . // <--- only one present.
> >
> inputs: <
>   key: "i2"
>   value: "n6"
> >
> outputs: <
>   key: "i0"
>   value: "n7"
> >
>   >
> >
> [...]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=131673&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131673
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:52
Start Date: 06/Aug/18 23:52
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #6161: [BEAM-3709] 
Portable combiner lifting support in py Dataflow Runner.
URL: https://github.com/apache/beam/pull/6161
 
 
   Modifies the way the CombineValues transform is communicated to
   Dataflow for FnApi pipelines to enable support for combiner lifting.
   
   **Important**: Don't merge this change in until #6128 is merged. Doing
   so will cause pipelines containing combines with side inputs to fail.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131673)
Time Spent: 1.5h  (was: 1h 20m)

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=131674&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131674
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:52
Start Date: 06/Aug/18 23:52
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #6161: [BEAM-3709] Portable 
combiner lifting support in py Dataflow Runner.
URL: https://github.com/apache/beam/pull/6161#issuecomment-410888451
 
 
   R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131674)
Time Spent: 1h 40m  (was: 1.5h)

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, sdk-py-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Python SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=131667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131667
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:31
Start Date: 06/Aug/18 23:31
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #6128: [BEAM-3709] Updating 
combine w/ side input translation in python.
URL: https://github.com/apache/beam/pull/6128#issuecomment-410885080
 
 
   R: @aaltay


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131667)
Time Spent: 1h 20m  (was: 1h 10m)

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, sdk-py-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Python SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131660
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:24
Start Date: 06/Aug/18 23:24
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #525: [BEAM-4761] Add 
Dataflow Runner
URL: https://github.com/apache/beam-site/pull/525
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/documentation/sdks/nexmark.md 
b/src/documentation/sdks/nexmark.md
index bf96803f29..6dac601f68 100644
--- a/src/documentation/sdks/nexmark.md
+++ b/src/documentation/sdks/nexmark.md
@@ -689,6 +689,8 @@ Each dashboard contains:
 
 [Nexmark performance spark 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712)
 
+[Nexmark performance dataflow 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5670405876482048)
+
 
 ### Output size dashboards links
 
@@ -698,3 +700,5 @@ Each dashboard contains:
 
 [Nexmark output size spark 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000)
 
+[Nexmark output size dataflow 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5647201107705856)
+


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131660)
Time Spent: 10h 10m  (was: 10h)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=131661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131661
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:24
Start Date: 06/Aug/18 23:24
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6160: 
[BEAM-5092] Make sure that SchemaCoder overrides consistentWithEquals.
URL: https://github.com/apache/beam/pull/6160#discussion_r208056593
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
 ##
 @@ -30,7 +30,9 @@
  *
  * This provider finds (recursively) all public getters and setters in a 
Java object, and creates
  * schemas and rows that bind to those fields. The field order in the schema 
is not guaranteed to
- * match the method order in the class.
+ * match the method order in the class. The Java object is expected to have 
implemented a correct
+ * .equals() method. TODO: Validate equals() method is provided, and if not 
generate a "slow" equals
+ * method based on * the schema.
 
 Review comment:
   Extra `*`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131661)
Time Spent: 20m  (was: 10m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit a763a756679965650b554d0c1a24a5b3630a78ef
Author: Mergebot 
AuthorDate: Mon Aug 6 23:23:55 2018 +

Prepare repository for deployment.
---
 content/documentation/sdks/java/nexmark/index.html | 4 
 1 file changed, 4 insertions(+)

diff --git a/content/documentation/sdks/java/nexmark/index.html 
b/content/documentation/sdks/java/nexmark/index.html
index 96349f1..9ef1dbe 100644
--- a/content/documentation/sdks/java/nexmark/index.html
+++ b/content/documentation/sdks/java/nexmark/index.html
@@ -1042,6 +1042,8 @@ PUBSUB_TOPCI=
 
 https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712";>Nexmark
 performance spark runner
 
+https://apache-beam-testing.appspot.com/explore?dashboard=5670405876482048";>Nexmark
 performance dataflow runner
+
 Output size dashboards links
 
 https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520";>Nexmark
 output size direct runner
@@ -1050,6 +1052,8 @@ PUBSUB_TOPCI=
 
 https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000";>Nexmark
 output size spark runner
 
+https://apache-beam-testing.appspot.com/explore?dashboard=5647201107705856";>Nexmark
 output size dataflow runner
+
 
   
 

[beam-site] branch asf-site updated (5050f9b -> a763a75)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 5050f9b  Prepare repository for deployment.
 add 5da4f0b  [BEAM-4761] Add Dataflow Runner
 add f64d2f0  This closes #525
 new a763a75  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/sdks/java/nexmark/index.html | 4 
 src/documentation/sdks/nexmark.md  | 4 
 2 files changed, 8 insertions(+)

[beam-site] 01/02: [BEAM-4761] Add Dataflow Runner

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 5da4f0bb064ce0c044e7cdad79dad59bac90730c
Author: Andrew Pilloud 
AuthorDate: Mon Aug 6 14:42:25 2018 -0700

[BEAM-4761] Add Dataflow Runner
---
 src/documentation/sdks/nexmark.md | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/documentation/sdks/nexmark.md 
b/src/documentation/sdks/nexmark.md
index bf96803..6dac601 100644
--- a/src/documentation/sdks/nexmark.md
+++ b/src/documentation/sdks/nexmark.md
@@ -689,6 +689,8 @@ Each dashboard contains:
 
 [Nexmark performance spark 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712)
 
+[Nexmark performance dataflow 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5670405876482048)
+
 
 ### Output size dashboards links
 
@@ -698,3 +700,5 @@ Each dashboard contains:
 
 [Nexmark output size spark 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000)
 
+[Nexmark output size dataflow 
runner](https://apache-beam-testing.appspot.com/explore?dashboard=5647201107705856)
+

[beam-site] 02/02: This closes #525

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f64d2f09ef15360554f4f5441e49ebb35fbc6344
Merge: 5050f9b 5da4f0b
Author: Mergebot 
AuthorDate: Mon Aug 6 23:20:57 2018 +

This closes #525

 src/documentation/sdks/nexmark.md | 4 
 1 file changed, 4 insertions(+)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131658&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131658
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:20
Start Date: 06/Aug/18 23:20
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #525: [BEAM-4761] Add 
Dataflow Runner
URL: https://github.com/apache/beam-site/pull/525#issuecomment-410883031
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131658)
Time Spent: 10h  (was: 9h 50m)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131657
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:19
Start Date: 06/Aug/18 23:19
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #525: [BEAM-4761] Add 
Dataflow Runner
URL: https://github.com/apache/beam-site/pull/525#issuecomment-410882899
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131657)
Time Spent: 9h 50m  (was: 9h 40m)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=131653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131653
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:15
Start Date: 06/Aug/18 23:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax opened a new pull request #6160: [BEAM-5092] 
Make sure that SchemaCoder overrides consistentWithEquals.
URL: https://github.com/apache/beam/pull/6160
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131653)
Time Spent: 10m
Remaining Estimate: 0h

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131651
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:11
Start Date: 06/Aug/18 23:11
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #525: [BEAM-4761] Add 
Dataflow Runner
URL: https://github.com/apache/beam-site/pull/525#issuecomment-410881372
 
 
   R: @akedin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131651)
Time Spent: 9h 40m  (was: 9.5h)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131650
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:09
Start Date: 06/Aug/18 23:09
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #525: [BEAM-4761] 
Add Dataflow Runner
URL: https://github.com/apache/beam-site/pull/525
 
 
   This adds the dataflow runner to the perf links.
   
   Once your pull request has been opened and assigned a number, please edit the
   URL below, replacing `PULL_REQUEST_NUMBER` with the number of your pull 
request.
   
   
http://apache-beam-website-pull-requests.storage.googleapis.com/PULL_REQUEST_NUMBER/index.html
   
   Finally, it will help us expedite review of your Pull Request if you tag
   someone (e.g. @username) to look at it.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131650)
Time Spent: 9.5h  (was: 9h 20m)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3809) beam_PerformanceTests_Python failing

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-3809.
--

> beam_PerformanceTests_Python failing
> 
>
> Key: BEAM-3809
> URL: https://issues.apache.org/jira/browse/BEAM-3809
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Looks like a misconfiguration based on the error.
>  
> Example test:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Python/1000/console]
>  
> Error I see:
> error: option --test-pipeline-options not recognized
> 2018-03-08 18:48:08,202 9e009d45 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 622, in RunBenchmark
> DoRunPhase(spec, collector, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 525, in DoRunPhase
> samples = spec.BenchmarkRun(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 159, in Run
> job_type=job_type)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 90, in SubmitJob
> assert retcode == 0, "Integration Test Failed."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3809) beam_PerformanceTests_Python failing

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu resolved BEAM-3809.

   Resolution: Fixed
Fix Version/s: Not applicable

> beam_PerformanceTests_Python failing
> 
>
> Key: BEAM-3809
> URL: https://issues.apache.org/jira/browse/BEAM-3809
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Looks like a misconfiguration based on the error.
>  
> Example test:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Python/1000/console]
>  
> Error I see:
> error: option --test-pipeline-options not recognized
> 2018-03-08 18:48:08,202 9e009d45 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 622, in RunBenchmark
> DoRunPhase(spec, collector, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 525, in DoRunPhase
> samples = spec.BenchmarkRun(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 159, in Run
> job_type=job_type)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 90, in SubmitJob
> assert retcode == 0, "Integration Test Failed."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131648
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 23:05
Start Date: 06/Aug/18 23:05
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #6141: [BEAM-4679] 
Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#issuecomment-410880263
 
 
   > Do we also need to add other primitive transforms here?
   Reference: 
https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java#L133
   
   I don't think that's necessary. This class only handles providers for roots 
of the pipeline. Theoretically only the ImpulseProvider should be necessary, 
but Java pipelines allow Flattens as roots, so to get the ValidatesRunner tests 
around that working this needed to be added.
   
   Basically, I'm trying to avoid adding anything unless the ValidatesRunner 
tests fail otherwise.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131648)
Time Spent: 1.5h  (was: 1h 20m)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131644&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131644
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:56
Start Date: 06/Aug/18 22:56
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6141: 
[BEAM-4679] Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#discussion_r208052819
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java
 ##
 @@ -33,6 +34,16 @@
  * based on the type of {@link PTransform} of the application.
  */
 class RootProviderRegistry {
+  public static RootProviderRegistry javaPortableRegistry(BundleFactory 
bundleFactory) {
+return new RootProviderRegistry(
+ImmutableMap.>builder()
+.put(
+IMPULSE_TRANSFORM_URN,
+new ImpulseEvaluatorFactory.ImpulseRootProvider(bundleFactory))
+.put(FLATTEN_TRANSFORM_URN, new EmptyInputProvider())
+.build());
+  }
+
   /** Returns a {@link RootProviderRegistry} that only supports the {@link 
Impulse} primitive. */
   public static RootProviderRegistry impulseRegistry(BundleFactory 
bundleFactory) {
 
 Review comment:
   Agreed. Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131644)
Time Spent: 1h 20m  (was: 1h 10m)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131643
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:44
Start Date: 06/Aug/18 22:44
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6141: 
[BEAM-4679] Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#discussion_r208050801
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java
 ##
 @@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct.portable;
+
+import java.util.Collection;
+import java.util.Collections;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+
+/** A {@link RootInputProvider} that provides no input bundles. */
+class EmptyInputProvider implements RootInputProvider {
+  EmptyInputProvider() {}
+
+  /**
+   * {@inheritDoc}.
+   *
+   * Returns an empty collection.
+   */
+  @Override
+  public Collection> getInitialInputs(
+  PTransformNode transform,
+  int targetParallelism) {
+return Collections.emptyList();
+  }
+}
 
 Review comment:
   Done. Spotless added it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131643)
Time Spent: 1h 10m  (was: 1h)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131641
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:40
Start Date: 06/Aug/18 22:40
Worklog Time Spent: 10m 
  Work Description: holdenk commented on a change in pull request #6157: 
[BEAM-5097] Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#discussion_r208050043
 
 

 ##
 File path: sdks/go/examples/cookbook/combine/combine.go
 ##
 @@ -63,11 +63,16 @@ type extractFn struct {
MinLength int `json:"min_length"`
 }
 
+// A global context for simplicity.
+var ctx = context.Background()
+
 func (f *extractFn) ProcessElement(row WordRow, emit func(string, string)) {
+   small_words := beam.NewCounter("example.namespace", "small_words")
 
 Review comment:
   Sounds good. @herohde pointed out the WC example uses counters too so I'm 
going to go take a look at that as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131641)
Time Spent: 1h  (was: 50m)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #793

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131642&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131642
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:40
Start Date: 06/Aug/18 22:40
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6157: [BEAM-5097][WIP] Add 
counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#issuecomment-410875484
 
 
   Switched to WIP since was pointed out another place uses counters that I 
didn't notice so I'll go check that as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131642)
Time Spent: 1h 10m  (was: 1h)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3878) Improve error reporting in calls.go

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3878?focusedWorklogId=131640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131640
 ]

ASF GitHub Bot logged work on BEAM-3878:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:39
Start Date: 06/Aug/18 22:39
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6156: [BEAM-3878] Improve 
error reporting in calls.go
URL: https://github.com/apache/beam/pull/6156#issuecomment-410875209
 
 
   cc @herohde thanks for your help with finding the specialize thing, do you 
have cycles to review this change?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131640)
Time Spent: 20m  (was: 10m)

> Improve error reporting in calls.go
> ---
>
> Key: BEAM-3878
> URL: https://issues.apache.org/jira/browse/BEAM-3878
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Bill Neubauer
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The error messages generated in calls.go are not as helpful as they could be.
> Instead of simply reporting "incompatible func type" it would be great if 
> they reported the topology of the actual function supplied versus what is 
> expected. That would make debugging a lot easier.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131636&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131636
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:35
Start Date: 06/Aug/18 22:35
Worklog Time Spent: 10m 
  Work Description: lostluck removed a comment on issue #6157: [BEAM-5097] 
Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#issuecomment-410874134
 
 
   Thanks for the PR!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131636)
Time Spent: 0.5h  (was: 20m)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131638&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131638
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:35
Start Date: 06/Aug/18 22:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6157: 
[BEAM-5097] Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#discussion_r208046722
 
 

 ##
 File path: sdks/go/examples/cookbook/combine/combine.go
 ##
 @@ -63,11 +63,16 @@ type extractFn struct {
MinLength int `json:"min_length"`
 }
 
+// A global context for simplicity.
+var ctx = context.Background()
 
 Review comment:
   Since this is a cookbook the correct way to get a context is to have it as 
the first argument (as per the go idiom) to the ProcessElement method. The beam 
runtime will automatically provide a context for the function during execution.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131638)
Time Spent: 50m  (was: 40m)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131637
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:35
Start Date: 06/Aug/18 22:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6157: 
[BEAM-5097] Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#discussion_r208048664
 
 

 ##
 File path: sdks/go/examples/cookbook/combine/combine.go
 ##
 @@ -63,11 +63,16 @@ type extractFn struct {
MinLength int `json:"min_length"`
 }
 
+// A global context for simplicity.
+var ctx = context.Background()
+
 func (f *extractFn) ProcessElement(row WordRow, emit func(string, string)) {
+   small_words := beam.NewCounter("example.namespace", "small_words")
 
 Review comment:
   For perf reasons it's better to declare the counter in the package scope, 
but either works correctly.
   
   Better to show the recommended path as part of real examples or recipes I 
think.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131637)
Time Spent: 40m  (was: 0.5h)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131635
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:34
Start Date: 06/Aug/18 22:34
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6157: [BEAM-5097] Add 
counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#issuecomment-410874134
 
 
   Thanks for the PR!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131635)
Time Spent: 20m  (was: 10m)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5092) Nexmark 10x performance regression

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570868#comment-16570868
 ] 

Andrew Pilloud commented on BEAM-5092:
--

Copy and paste from `job_PostCommit_Java_Nexmark_Direct.groovy`?

 
{code:java}
 ./gradlew :beam-sdks-java-nexmark:run 
-Pnexmark.runner=":beam-runners-direct-java" 
-Pnexmark.args="–runner=DirectRunner --streaming=false --suite=SMOKE 
--manageResources=false --monitorJobs=true --enforceEncodability=true 
--enforceImmutability=true"
{code}
 

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4780) Entry point for ULR JobService compatible with TestPortableRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4780?focusedWorklogId=131632&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131632
 ]

ASF GitHub Bot logged work on BEAM-4780:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:32
Start Date: 06/Aug/18 22:32
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6151: 
[BEAM-4780] Updating to DockerJobBundleFactory in ReferenceRunner.
URL: https://github.com/apache/beam/pull/6151#discussion_r208048735
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
 ##
 @@ -168,6 +170,7 @@ public void execute() throws Exception {
 .setWorkerId("foo")
 .setResourceLimits(Resources.getDefaultInstance())
 .build();
+JobInfo jobInfo = JobInfo.create("id", "reference", "retrieval-token", 
options);
 
 Review comment:
   Good point. I think it's a bit outside the scope of this PR, but I wrote a 
bug and added a TODO so I can look into it later.
   
   Since this PR right now is mainly for trying to get #6073 working I don't 
really want to spend too long working on the JobInfo yet (although I will 
almost certainly need to get to it at some point).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131632)
Time Spent: 40m  (was: 0.5h)

> Entry point for ULR JobService compatible with TestPortableRunner
> -
>
> Key: BEAM-4780
> URL: https://issues.apache.org/jira/browse/BEAM-4780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/pull/5935] introduces a TestPortableRunner 
> that can run ValidatesRunner tests against a given portable runner, 
> identified by a class name of a shim that can start/stop its JobService 
> endpoint.
> For ULR to run VR tests, it needs to provide such a shim.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5092) Nexmark 10x performance regression

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570860#comment-16570860
 ] 

Reuven Lax commented on BEAM-5092:
--

Is there an easy way to manually run the benchmark to verify fixes?

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5092) Nexmark 10x performance regression

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570854#comment-16570854
 ] 

Andrew Pilloud edited comment on BEAM-5092 at 8/6/18 10:16 PM:
---

It looks like Query 12 is broken in Streaming mode, I suspect it is related. 
Hopefully whatever you change to fix the performance will also fix this.

https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520


was (Author: apilloud):
It looks like Query 12 is broken in Streaming mode, I suspect it is related. 
Hopefully whatever you change to fix the performance will also fix this.

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5092) Nexmark 10x performance regression

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570854#comment-16570854
 ] 

Andrew Pilloud commented on BEAM-5092:
--

It looks like Query 12 is broken in Streaming mode, I suspect it is related. 
Hopefully whatever you change to fix the performance will also fix this.

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=131630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131630
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:14
Start Date: 06/Aug/18 22:14
Worklog Time Spent: 10m 
  Work Description: holdenk opened a new pull request #6157: [BEAM-5097] 
Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157
 
 
   Add counter to combine example in go sdk to demonstrate counters.
   I'm basing this off of the test because there aren't any examples yet. I'm 
especially curious if this is the right thing to do with the context cc 
@herohde @lostluck 
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131630)
Time Spent: 10m
Remaining Estimate: 0h

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk

[jira] [Created] (BEAM-5097) Increment counter for "small words" in go SDK example

[holdenk (JIRA)]

holdenk created BEAM-5097:
-

 Summary: Increment counter for "small words" in go SDK example
 Key: BEAM-5097
 URL: https://issues.apache.org/jira/browse/BEAM-5097
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: holdenk
Assignee: holdenk


Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5092) Nexmark 10x performance regression

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570847#comment-16570847
 ] 

Reuven Lax commented on BEAM-5092:
--

Looking at the PR there are only two things that could have affected Nexmark:

   1. I changed various longs to DateTime objects. This might be slower, but is 
a good change here.

   2. I added a schema to Nexmark types for SQL nexmark, but looks like that is 
taking priority over the hard-coded Nexmark Coder. This is easily fixed.

Also, SchemaCoder isn't expected to be slow, but on investigation it appears 
that SchemaCoder does not implement structuralValue. This means that Beam will 
encode elements every time it want to check them for equality, which is quite 
expensive. This is also easy to fix.

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3878) Improve error reporting in calls.go

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3878?focusedWorklogId=131623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131623
 ]

ASF GitHub Bot logged work on BEAM-3878:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:07
Start Date: 06/Aug/18 22:07
Worklog Time Spent: 10m 
  Work Description: holdenk opened a new pull request #6156: [BEAM-3878] 
Improve error reporting in calls.go
URL: https://github.com/apache/beam/pull/6156
 
 
   Improve error reporting in calls.go by outputing the expected and provided 
number of arguments when a mismatch occurs.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ X ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131623)
Time Spent: 10m
Remaining Estimate: 0h

> Improve error reporting in calls.go
> ---
>
> Key: BEAM-3878
> URL: https://issues.apache.org/jira/browse/BEAM-3878
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Bill Neubauer
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The error messages generated in calls.go are not as helpful as th

[jira] [Work logged] (BEAM-5064) Support ORDER BY in Global window

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5064?focusedWorklogId=131624&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131624
 ]

ASF GitHub Bot logged work on BEAM-5064:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:07
Start Date: 06/Aug/18 22:07
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6154: 
[BEAM-5064][SQL] Support Order By in Global Window
URL: https://github.com/apache/beam/pull/6154#issuecomment-410868315
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131624)
Time Spent: 40m  (was: 0.5h)

> Support ORDER BY in Global window
> -
>
> Key: BEAM-5064
> URL: https://issues.apache.org/jira/browse/BEAM-5064
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Beam does not support `ORDER BY` in global window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5064) Support ORDER BY in Global window

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5064?focusedWorklogId=131622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131622
 ]

ASF GitHub Bot logged work on BEAM-5064:


Author: ASF GitHub Bot
Created on: 06/Aug/18 22:06
Start Date: 06/Aug/18 22:06
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6154: [BEAM-5064][SQL] 
Support Order By in Global Window
URL: https://github.com/apache/beam/pull/6154#issuecomment-410868315
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131622)
Time Spent: 0.5h  (was: 20m)

> Support ORDER BY in Global window
> -
>
> Key: BEAM-5064
> URL: https://issues.apache.org/jira/browse/BEAM-5064
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Beam does not support `ORDER BY` in global window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5094) go generate specialize isn't super clear where to find specialize

[holdenk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

holdenk closed BEAM-5094.
-
   Resolution: Not A Problem
Fix Version/s: Not applicable

nvm, was in sdk/go/cmd but I was looking in sdk/go/pkg

> go generate specialize isn't super clear where to find specialize
> -
>
> Key: BEAM-5094
> URL: https://issues.apache.org/jira/browse/BEAM-5094
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: Henning Rohde
>Priority: Trivial
> Fix For: Not applicable
>
>
> If I try and update the templates go generate tells me the specialize command 
> is not found. Searching online we seem to be the only project which uses it ( 
> [https://www.google.com/search?ei=m7xoW9TpJY6O_wS1k4WoCQ&q=%22go%3Agenerate+specialize%22&oq=%22go%3Agenerate+specialize%22&gs_l=psy-ab.3...703840.705307.0.705535.3.3.0.0.0.0.50.139.3.3.00...1c.1.64.psy-ab..0.0.00.oe1VY3mk99c]
>  ) so I'm not entirely sure where to get specialize from. Is it a perhaps a 
> tool that we forgot to check in or is it a build artifact that I need to put 
> in my path?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5096) Create nightly snapshot for Beam Python SDK

[yifan zou (JIRA)]

yifan zou created BEAM-5096:
---

 Summary: Create nightly snapshot for Beam Python SDK
 Key: BEAM-5096
 URL: https://issues.apache.org/jira/browse/BEAM-5096
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: yifan zou
Assignee: yifan zou






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4844) Test Auto JIRA Subtask 000

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4844.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Test Auto JIRA Subtask 000
> --
>
> Key: BEAM-4844
> URL: https://issues.apache.org/jira/browse/BEAM-4844
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #5680

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-5092) Nexmark 10x performance regression

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570826#comment-16570826
 ] 

Reuven Lax commented on BEAM-5092:
--

Do we know the range of PRs where this happened? There are a couple of 
possibilities here, some of them expected (I changed some thing to be more 
strongly typed, which might result in some performance slowdown).

 

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5095) Fill out JobInfo properly in ReferenceRunner

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5095:
-

 Summary: Fill out JobInfo properly in ReferenceRunner
 Key: BEAM-5095
 URL: https://issues.apache.org/jira/browse/BEAM-5095
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


In the ReferenceRunner class JobInfo is currently filled out with several 
placeholder values, particularly for job ID and retrieval token. This bug is 
for filling out these entries with values that aren't placeholder values, such 
as actual unique job IDs and an actual retrieval token.

Alternatively, if it happens that the Reference Runner doesn't actually need 
any of the placeholder values filled in to function, and adding them for 
completeness' sake is inordinately hard, this bug can be marked obsolete.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5074) Make BeamSqlRowComparator more general

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5074.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

Consolidate this Jira with BEAM-5064

> Make BeamSqlRowComparator more general
> --
>
> Key: BEAM-5074
> URL: https://issues.apache.org/jira/browse/BEAM-5074
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> BeamSqlRowComparator returns reversed comparison results (if a < b in terms 
> of value, BeamSqlRowComparator returns a > b). 
>  
> Add a flag to turn on/off this reversed effect. The reason of doing this is 
> to help implement ORDER BY ASC/DESC in the normal way.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3878) Improve error reporting in calls.go

[holdenk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570815#comment-16570815
 ] 

holdenk commented on BEAM-3878:
---

I started working on this, but then I ran into some confusion with the go 
generate command we are using (see 
[https://jira.apache.org/jira/browse/BEAM-5094] ).

> Improve error reporting in calls.go
> ---
>
> Key: BEAM-3878
> URL: https://issues.apache.org/jira/browse/BEAM-3878
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Bill Neubauer
>Priority: Minor
>
> The error messages generated in calls.go are not as helpful as they could be.
> Instead of simply reporting "incompatible func type" it would be great if 
> they reported the topology of the actual function supplied versus what is 
> expected. That would make debugging a lot easier.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1189

[Apache Jenkins Server]

See 


Changes:

[markliu] Upgrade mockito-core to 1.10.19

--
[...truncated 12.81 MB...]
  BeamIOSourceRel(table=[[beam, message]])


org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT > 
testSQLTypes STANDARD_ERROR
Aug 06, 2018 9:37:05 PM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4 
getTempFilePrefix
INFO: Writing BigQuery temporary files to 
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8/
 before loading them.
Aug 06, 2018 9:37:05 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter 

INFO: Opening TableRowWriter to 
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8/97c12509-83df-4c7b-bd8b-ee66c217efbf.
Aug 06, 2018 9:37:06 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 1 files into {datasetId=integ_test, 
projectId=apache-beam-testing, 
tableId=BigQueryReadWriteIT_testSQLTypes_2018_08_06_21_36_59_653_6356496124533583493}
 using job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 location=US, projectId=apache-beam-testing}, attempt 1
Aug 06, 2018 9:37:06 PM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 location=US, projectId=apache-beam-testing}.
bq show -j --format=prettyjson --project_id=apache-beam-testing 
beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0
Aug 06, 2018 9:37:06 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 location=US, projectId=apache-beam-testing} started
Aug 06, 2018 9:37:06 PM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job 
beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing 
beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0

org.apache.beam.sdk.extensions.sql.PubsubToBigqueryIT > testSimpleInsert 
STANDARD_ERROR
Aug 06, 2018 9:37:06 PM 
org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource createRandomSubscription
WARNING: Created subscription 
projects/apache-beam-testing/subscriptions/integ-test-PubsubToBigqueryIT-testSimpleInsert-2018-08-06-21-37-00-223-events-5348202260404966627_beam_-2221412512525908592
 to topic 
projects/apache-beam-testing/topics/integ-test-PubsubToBigqueryIT-testSimpleInsert-2018-08-06-21-37-00-223-events-5348202260404966627.
 Note this subscription WILL NOT be deleted when the pipeline terminates

org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT > 
testSQLTypes STANDARD_ERROR
Aug 06, 2018 9:37:07 PM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job 
beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing 
beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0
Aug 06, 2018 9:37:09 PM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 location=US, projectId=apache-beam-testing} completed in state DONE
Aug 06, 2018 9:37:09 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins080621370534f0615b_88f91459cdf44cc4a92a4ae323e6e4f8_4c943dd1065e85f12a3055a4eabbc8cb_1_0-0,
 location=US, projectId=apache-beam-testing} succeeded. Statistics: 
{"creationTime":"1533591426220","endTime":"1533591427544","load":{"badRecords":"0","inputFile

[jira] [Work logged] (BEAM-5064) Support ORDER BY in Global window

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5064?focusedWorklogId=131606&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131606
 ]

ASF GitHub Bot logged work on BEAM-5064:


Author: ASF GitHub Bot
Created on: 06/Aug/18 21:38
Start Date: 06/Aug/18 21:38
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6154: [BEAM-5064][SQL] 
Support order by in global window
URL: https://github.com/apache/beam/pull/6154#issuecomment-410861283
 
 
   R: @XuMingmin @apilloud @akedin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131606)
Time Spent: 20m  (was: 10m)

> Support ORDER BY in Global window
> -
>
> Key: BEAM-5064
> URL: https://issues.apache.org/jira/browse/BEAM-5064
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Beam does not support `ORDER BY` in global window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5064) Support ORDER BY in Global window

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5064?focusedWorklogId=131605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131605
 ]

ASF GitHub Bot logged work on BEAM-5064:


Author: ASF GitHub Bot
Created on: 06/Aug/18 21:38
Start Date: 06/Aug/18 21:38
Worklog Time Spent: 10m 
  Work Description: amaliujia opened a new pull request #6154: 
[BEAM-5064][SQL] Support order by in global window
URL: https://github.com/apache/beam/pull/6154
 
 
   Support Order by in global window.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131605)
Time Spent: 10m
Remaining Estimate: 0h

> Support ORDER BY in Global window
> -
>
> Key: BEAM-5064
> URL: https://issues.apache.org/jira/browse/BEAM-5064
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Beam does not support `ORDER BY` in global window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5094) go generate specialize isn't super clear where to find specialize

[holdenk (JIRA)]

holdenk created BEAM-5094:
-

 Summary: go generate specialize isn't super clear where to find 
specialize
 Key: BEAM-5094
 URL: https://issues.apache.org/jira/browse/BEAM-5094
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: holdenk
Assignee: Henning Rohde


If I try and update the templates go generate tells me the specialize command 
is not found. Searching online we seem to be the only project which uses it ( 
[https://www.google.com/search?ei=m7xoW9TpJY6O_wS1k4WoCQ&q=%22go%3Agenerate+specialize%22&oq=%22go%3Agenerate+specialize%22&gs_l=psy-ab.3...703840.705307.0.705535.3.3.0.0.0.0.50.139.3.3.00...1c.1.64.psy-ab..0.0.00.oe1VY3mk99c]
 ) so I'm not entirely sure where to get specialize from. Is it a perhaps a 
tool that we forgot to check in or is it a build artifact that I need to put in 
my path?

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3230) beam_PerformanceTests_Python is failing

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-3230.
--

> beam_PerformanceTests_Python is failing
> ---
>
> Key: BEAM-3230
> URL: https://issues.apache.org/jira/browse/BEAM-3230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Mark Liu
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkings beam_PerformanceTests_Python stage is failing for python builds.
> Here is the link to a failure console output 
> https://builds.apache.org/job/beam_PerformanceTests_Python/582/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3230) beam_PerformanceTests_Python is failing

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570810#comment-16570810
 ] 

Mark Liu commented on BEAM-3230:


Fix is merged. Latest schedule build passed: 
https://builds.apache.org/job/beam_PerformanceTests_Python/1272/

Will close this ticket.

> beam_PerformanceTests_Python is failing
> ---
>
> Key: BEAM-3230
> URL: https://issues.apache.org/jira/browse/BEAM-3230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkings beam_PerformanceTests_Python stage is failing for python builds.
> Here is the link to a failure console output 
> https://builds.apache.org/job/beam_PerformanceTests_Python/582/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3230) beam_PerformanceTests_Python is failing

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu resolved BEAM-3230.

   Resolution: Fixed
Fix Version/s: Not applicable

> beam_PerformanceTests_Python is failing
> ---
>
> Key: BEAM-3230
> URL: https://issues.apache.org/jira/browse/BEAM-3230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Mark Liu
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkings beam_PerformanceTests_Python stage is failing for python builds.
> Here is the link to a failure console output 
> https://builds.apache.org/job/beam_PerformanceTests_Python/582/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3230) beam_PerformanceTests_Python is failing

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu reassigned BEAM-3230:
--

Assignee: Mark Liu  (was: Ankur Goenka)

> beam_PerformanceTests_Python is failing
> ---
>
> Key: BEAM-3230
> URL: https://issues.apache.org/jira/browse/BEAM-3230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Mark Liu
>Priority: Critical
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkings beam_PerformanceTests_Python stage is failing for python builds.
> Here is the link to a failure console output 
> https://builds.apache.org/job/beam_PerformanceTests_Python/582/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3230) beam_PerformanceTests_Python is failing

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570809#comment-16570809
 ] 

Mark Liu commented on BEAM-3230:


https://github.com/apache/beam/pull/6142 will do the fix

> beam_PerformanceTests_Python is failing
> ---
>
> Key: BEAM-3230
> URL: https://issues.apache.org/jira/browse/BEAM-3230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Critical
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkings beam_PerformanceTests_Python stage is failing for python builds.
> Here is the link to a failure console output 
> https://builds.apache.org/job/beam_PerformanceTests_Python/582/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5093) Get ReferenceRunner to a state suitable for general use.

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5093:
-

 Summary: Get ReferenceRunner to a state suitable for general use.
 Key: BEAM-5093
 URL: https://issues.apache.org/jira/browse/BEAM-5093
 Project: Beam
  Issue Type: New Feature
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


The Reference Runner currently cannot be used to run a pipeline directly, does 
not have adequate testing, and cannot run all pipelines properly. This is a 
general bug for holding issues that need to be fixed before the Reference 
Runner can be considered ready for general use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1188

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5041) Java Fn SDK Harness skips unprocessed pCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5041?focusedWorklogId=131599&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131599
 ]

ASF GitHub Bot logged work on BEAM-5041:


Author: ASF GitHub Bot
Created on: 06/Aug/18 21:21
Start Date: 06/Aug/18 21:21
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6093: [BEAM-5041] Java Fn 
SDK Harness use pTransform to track processed graph
URL: https://github.com/apache/beam/pull/6093#issuecomment-410856782
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131599)
Time Spent: 2h 10m  (was: 2h)

> Java Fn SDK Harness skips unprocessed pCollections
> --
>
> Key: BEAM-5041
> URL: https://issues.apache.org/jira/browse/BEAM-5041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Java Sdk Harness used pCollections to keep track of computed consumers 
> [here|https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java#L158].
>  This is incorrect as consumers are based on pTransforms so pTransforms 
> should be used to keep track of computed consumers.
> In case of Flatten, this creates an issue where pTransforms having same input 
> as that to flatten are not executed. This causes 
> [https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java#L316]
>  to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5679

[Apache Jenkins Server]

See 


Changes:

[amaliujia] remove RelCollationTraitDef

--
[...truncated 823.79 KB...]
test_type_check_invalid_composite_type 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_be_tuple 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_have_same_arity 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_duplicates_ignored 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param_cant_be_object_instance 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_nested_unions_flattened 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_nested_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_composite_type_in_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_not_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_repr (apache_beam.typehints.typehints_test.UnionHintTestCase) 
... ok
test_deprecated_with_since_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_with_since_current_message 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_since_should_fail 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_with_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_with_current_message 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
Tests that the filter 'once' is sufficient to print once per ... ok
test_equal_objects (apache_beam.utils.counters_test.CounterNameTest) ... ok
test_hash_two_objects (apache_beam.utils.counters_test.CounterNameTest) ... ok
test_method_forwarding_not_windows (apache_beam.utils.processes_test.Exec) ... 
ok
test_method_forwarding_windows (apache_beam.utils.processes_test.Exec) ... ok
test_call_two_objects (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_single_failure (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_two_failures (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_log_calls_for_permanent_failure (apache_beam.utils.retry_test.RetryTest) 
... ok
test_log_calls_for_transient_failure (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_default_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_explicit_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_initial_delay (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_http_error_that_should_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... SKIP: google-apitools is not 
installed
test_with_http_error_that_should_not_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... SKIP: google-apitools is not 
installed
test_with_no_retry_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_real_clock (apache_beam.utils.retry_test.RetryTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_of (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_precision (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_sort_order (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_str (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_from_rfc3339 (apache_beam.utils.timestamp_tes

[jira] [Work logged] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1009?focusedWorklogId=131588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131588
 ]

ASF GitHub Bot logged work on BEAM-1009:


Author: ASF GitHub Bot
Created on: 06/Aug/18 21:05
Start Date: 06/Aug/18 21:05
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6153: [BEAM-1009] Upgrade 
mockito-core to 1.10.19
URL: https://github.com/apache/beam/pull/6153
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 05c0636b240..e67c8369e52 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -403,7 +403,7 @@ class BeamModulePlugin implements Plugin {
 kafka_2_11  : 
"org.apache.kafka:kafka_2.11:$kafka_version",
 kafka_clients   : 
"org.apache.kafka:kafka-clients:$kafka_version",
 malhar_library  : 
"org.apache.apex:malhar-library:$apex_malhar_version",
-mockito_core: 
"org.mockito:mockito-core:1.9.5",
+mockito_core: 
"org.mockito:mockito-core:1.10.19",
 netty_handler   : 
"io.netty:netty-handler:$netty_version",
 netty_tcnative_boringssl_static : 
"io.netty:netty-tcnative-boringssl-static:2.0.8.Final",
 netty_transport_native_epoll: 
"io.netty:netty-transport-native-epoll:$netty_version",


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131588)
Time Spent: 10m
Remaining Estimate: 0h

> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (6429412 -> abb3d34)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6429412  Merge pull request #6139 from 
amaliujia/rui_wang-remove_RelCollationTraitDef
 add e0bce46  Upgrade mockito-core to 1.10.19
 new abb3d34  [BEAM-1009] Upgrade mockito-core to 1.10.19

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] 01/01: [BEAM-1009] Upgrade mockito-core to 1.10.19

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit abb3d348deca8859f9d15ceb57c86483441cef94
Merge: 6429412 e0bce46
Author: Lukasz Cwik 
AuthorDate: Mon Aug 6 14:05:08 2018 -0700

[BEAM-1009] Upgrade mockito-core to 1.10.19

 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Assigned] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-1009:
---

Assignee: (was: Pei He)

> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Major
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4780) Entry point for ULR JobService compatible with TestPortableRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4780?focusedWorklogId=131587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131587
 ]

ASF GitHub Bot logged work on BEAM-4780:


Author: ASF GitHub Bot
Created on: 06/Aug/18 21:03
Start Date: 06/Aug/18 21:03
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #6151: 
[BEAM-4780] Updating to DockerJobBundleFactory in ReferenceRunner.
URL: https://github.com/apache/beam/pull/6151#discussion_r208028663
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
 ##
 @@ -168,6 +170,7 @@ public void execute() throws Exception {
 .setWorkerId("foo")
 .setResourceLimits(Resources.getDefaultInstance())
 .build();
+JobInfo jobInfo = JobInfo.create("id", "reference", "retrieval-token", 
options);
 
 Review comment:
   Should the reference runner allow for multiple (concurrent?) jobs? If so, we 
may need to actually generate unique job ids here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131587)
Time Spent: 0.5h  (was: 20m)

> Entry point for ULR JobService compatible with TestPortableRunner
> -
>
> Key: BEAM-4780
> URL: https://issues.apache.org/jira/browse/BEAM-4780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/pull/5935] introduces a TestPortableRunner 
> that can run ValidatesRunner tests against a given portable runner, 
> identified by a class name of a shim that can start/stop its JobService 
> endpoint.
> For ULR to run VR tests, it needs to provide such a shim.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5072) SortRelNode is wrongly removed

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5072.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> SortRelNode is wrongly removed
> --
>
> Key: BEAM-5072
> URL: https://issues.apache.org/jira/browse/BEAM-5072
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> ORDER BY query generates LogicalSort in logical plan, but it will be removed 
> wrongly during optimization by the SortRemoveRule



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5092) Nexmark 10x performance regression

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud updated BEAM-5092:
-
Description: 
There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
jobs. It first showed up in this build:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]

[https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]

[https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]

  was:
There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
jobs. It first showed up in this build:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]

[https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384

|https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]


> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5092) Nexmark 10x performance regression

[Andrew Pilloud (JIRA)]

Andrew Pilloud created BEAM-5092:


 Summary: Nexmark 10x performance regression
 Key: BEAM-5092
 URL: https://issues.apache.org/jira/browse/BEAM-5092
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Andrew Pilloud
Assignee: Reuven Lax


There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
jobs. It first showed up in this build:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]

[https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384

|https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5041) Java Fn SDK Harness skips unprocessed pCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5041?focusedWorklogId=131580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131580
 ]

ASF GitHub Bot logged work on BEAM-5041:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:45
Start Date: 06/Aug/18 20:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6093: [BEAM-5041] Java Fn 
SDK Harness use pTransform to track processed graph
URL: https://github.com/apache/beam/pull/6093#issuecomment-410847365
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131580)
Time Spent: 2h  (was: 1h 50m)

> Java Fn SDK Harness skips unprocessed pCollections
> --
>
> Key: BEAM-5041
> URL: https://issues.apache.org/jira/browse/BEAM-5041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Java Sdk Harness used pCollections to keep track of computed consumers 
> [here|https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java#L158].
>  This is incorrect as consumers are based on pTransforms so pTransforms 
> should be used to keep track of computed consumers.
> In case of Flatten, this creates an issue where pTransforms having same input 
> as that to flatten are not executed. This causes 
> [https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java#L316]
>  to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5069) Incorrect string type check in _OutputProcessor.process_outputs.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5069:
--
Affects Version/s: (was: 2.6.0)

> Incorrect string type check in _OutputProcessor.process_outputs.
> 
>
> Key: BEAM-5069
> URL: https://issues.apache.org/jira/browse/BEAM-5069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
>
> Due to _# cython: language_level=3_ directive in 
> [common.py|https://github.com/apache/beam/blob/8f1d86ea7bd5b118bc7e638d4d484445101526a1/sdks/python/apache_beam/runners/common.py#L18],
>  the type check in 
> [common.py|https://github.com/apache/beam/blob/8f1d86ea7bd5b118bc7e638d4d484445101526a1/sdks/python/apache_beam/runners/common.py#L729]
>  does not recognize Python 2 strings, which have type 'bytes'.
> This causes pipelines to crash when this codepath is executed in Cythonized 
> version of the SDK.
> The regression happened during ongoing efforts to make Beam codebase Python 3 
> compatible.
> The fix will need to be cherry-picked into the 2.6.0 release.
> Working on the fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5069) Incorrect string type check in _OutputProcessor.process_outputs.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570749#comment-16570749
 ] 

Valentyn Tymofieiev commented on BEAM-5069:
---

Fix has been CP'ed in the release: https://github.com/apache/beam/pull/6138

> Incorrect string type check in _OutputProcessor.process_outputs.
> 
>
> Key: BEAM-5069
> URL: https://issues.apache.org/jira/browse/BEAM-5069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
>
> Due to _# cython: language_level=3_ directive in 
> [common.py|https://github.com/apache/beam/blob/8f1d86ea7bd5b118bc7e638d4d484445101526a1/sdks/python/apache_beam/runners/common.py#L18],
>  the type check in 
> [common.py|https://github.com/apache/beam/blob/8f1d86ea7bd5b118bc7e638d4d484445101526a1/sdks/python/apache_beam/runners/common.py#L729]
>  does not recognize Python 2 strings, which have type 'bytes'.
> This causes pipelines to crash when this codepath is executed in Cythonized 
> version of the SDK.
> The regression happened during ongoing efforts to make Beam codebase Python 3 
> compatible.
> The fix will need to be cherry-picked into the 2.6.0 release.
> Working on the fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4726) Reduce ParDo per element Invoke overhead

[Robert Burke (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570753#comment-16570753
 ] 

Robert Burke commented on BEAM-4726:


It might be possible to further reduce the hot path per element and remove the 
conditional checks on inputs, such as extra arguments (like emitters & side 
inputs), and handling outputs, as are consistent per invocation.

 

Except for error handling, everything is identical between invocations, so it 
might be possible to target the specific assignments into the args, and avoid 
repeating values that aren't changing (eg, the context, the emitters & side 
input iterators, which only need to be re-set.)

> Reduce ParDo per element Invoke overhead
> 
>
> Key: BEAM-4726
> URL: https://issues.apache.org/jira/browse/BEAM-4726
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> Each call to invoke allocates a new args []interface{}, but the common case 
> is to run the same ProcessElement function over and again. It should be 
> possible to have a container struct to retain the args slice, and avoid 
> recomputing the indices for where to assign parameters before calling the 
> ProcessElementFn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5091) Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test suite.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570748#comment-16570748
 ] 

Valentyn Tymofieiev commented on BEAM-5091:
---

Currently, this is blocked by BEAM-5089.

> Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test 
> suite.
> -
>
> Key: BEAM-5091
> URL: https://issues.apache.org/jira/browse/BEAM-5091
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4727) Reduce metrics overhead

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-4727.

   Resolution: Fixed
Fix Version/s: 2.7.0

It may be possible to reduce this even further, but to me it's not immediately 
obvious how without also changing the API. The suggested improvements have also 
been implemented. If there's additional opportunity, feel free to re-open.

> Reduce metrics overhead
> ---
>
> Key: BEAM-4727
> URL: https://issues.apache.org/jira/browse/BEAM-4727
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> There are a few opportunities to avoid metrics overhead.
> First when setting state in the context, we allocate a new one for the stored 
> value, per element. This generates a fair amount of objects for the garbage 
> collector too. If we retain and re-use contexts within a bundle, we would 
> have the opportunity to save on these costs.
> Also, it's possible that we have overhead on the metric updating paths. We 
> can possibly do better than the general sync.Map, and avoid the type 
> assertion cost for extracting values of known types from the maps.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5030) Consolidate defer overhead per bundle

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke reassigned BEAM-5030:
--

Assignee: holdenk  (was: Robert Burke)

> Consolidate defer overhead per bundle
> -
>
> Key: BEAM-5030
> URL: https://issues.apache.org/jira/browse/BEAM-5030
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: holdenk
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> At present, reflectx.CallNoPanic is invoked for every element, and adds  
> recovery defers for each element. Defers are cheap, but not "per element" 
> cheap. In a simple pipeline (a single ParDo), this represents around ~3% 
> overhead for the pipeline, which is aggravated for every pardo layer.
> Since the beam model's unit of re-triable computation is per bundle, the 
> recovery defers should be invoked per bundle only at the root. This is 
> already being done in 
> [exec/plan.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/plan.go#L112]
> This could be as simple as calling Call directly in 
> [exec/fn.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fn.go#L99]
>  instead of with the reflectx wrapper as it is presently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5030) Consolidate defer overhead per bundle

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-5030.

   Resolution: Fixed
Fix Version/s: 2.7.0

> Consolidate defer overhead per bundle
> -
>
> Key: BEAM-5030
> URL: https://issues.apache.org/jira/browse/BEAM-5030
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: holdenk
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> At present, reflectx.CallNoPanic is invoked for every element, and adds  
> recovery defers for each element. Defers are cheap, but not "per element" 
> cheap. In a simple pipeline (a single ParDo), this represents around ~3% 
> overhead for the pipeline, which is aggravated for every pardo layer.
> Since the beam model's unit of re-triable computation is per bundle, the 
> recovery defers should be invoked per bundle only at the root. This is 
> already being done in 
> [exec/plan.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/plan.go#L112]
> This could be as simple as calling Call directly in 
> [exec/fn.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fn.go#L99]
>  instead of with the reflectx wrapper as it is presently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-4276.

   Resolution: Fixed
Fix Version/s: 2.7.0

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131577
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:37
Start Date: 06/Aug/18 20:37
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6141: [BEAM-4679] 
Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#issuecomment-410845152
 
 
   Please rerun `./gradlew  spotlessApply`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131577)
Time Spent: 1h  (was: 50m)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131575
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:35
Start Date: 06/Aug/18 20:35
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6141: 
[BEAM-4679] Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#discussion_r208020819
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java
 ##
 @@ -33,6 +34,16 @@
  * based on the type of {@link PTransform} of the application.
  */
 class RootProviderRegistry {
+  public static RootProviderRegistry javaPortableRegistry(BundleFactory 
bundleFactory) {
+return new RootProviderRegistry(
+ImmutableMap.>builder()
+.put(
+IMPULSE_TRANSFORM_URN,
+new ImpulseEvaluatorFactory.ImpulseRootProvider(bundleFactory))
+.put(FLATTEN_TRANSFORM_URN, new EmptyInputProvider())
+.build());
+  }
+
   /** Returns a {@link RootProviderRegistry} that only supports the {@link 
Impulse} primitive. */
   public static RootProviderRegistry impulseRegistry(BundleFactory 
bundleFactory) {
 
 Review comment:
   Shall we remove impulse registry as it is not used anymore?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131575)
Time Spent: 50m  (was: 40m)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4679) Support portable combiner lifting in Java Reference Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4679?focusedWorklogId=131574&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131574
 ]

ASF GitHub Bot logged work on BEAM-4679:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:35
Start Date: 06/Aug/18 20:35
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6141: 
[BEAM-4679] Re-adding Flatten roots in ULR.
URL: https://github.com/apache/beam/pull/6141#discussion_r208020590
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java
 ##
 @@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct.portable;
+
+import java.util.Collection;
+import java.util.Collections;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+
+/** A {@link RootInputProvider} that provides no input bundles. */
+class EmptyInputProvider implements RootInputProvider {
+  EmptyInputProvider() {}
+
+  /**
+   * {@inheritDoc}.
+   *
+   * Returns an empty collection.
+   */
+  @Override
+  public Collection> getInitialInputs(
+  PTransformNode transform,
+  int targetParallelism) {
+return Collections.emptyList();
+  }
+}
 
 Review comment:
   Please add an empty line


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131574)
Time Spent: 50m  (was: 40m)

> Support portable combiner lifting in Java Reference Runner
> --
>
> Key: BEAM-4679
> URL: https://issues.apache.org/jira/browse/BEAM-4679
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Adjust the Reference Runner in Java to support portable combiner lifting as 
> described in the following doc:
> https://s.apache.org/beam-runner-api-combine-model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=131572&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131572
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:32
Start Date: 06/Aug/18 20:32
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r208020210
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/pubsub.py
 ##
 @@ -33,8 +33,9 @@
 from __future__ import absolute_import
 
 import re
+from builtins import object
 
-from six import text_type
+from past.builtins import basestring
 
 Review comment:
   https://github.com/apache/beam/pull/6144 cleans this up here as well as in 
the place you mentioned, since we have reverted `# cython: language_level=3`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131572)
Time Spent: 6h 50m  (was: 6h 40m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=131571&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131571
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:30
Start Date: 06/Aug/18 20:30
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6149: [BEAM-4761] Pin 
Dataflow Nexmark Autoscale to 4
URL: https://github.com/apache/beam/pull/6149#issuecomment-410843226
 
 
   
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Dataflow/101/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131571)
Time Spent: 9h 20m  (was: 9h 10m)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5074) Make BeamSqlRowComparator more general

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5074?focusedWorklogId=131570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131570
 ]

ASF GitHub Bot logged work on BEAM-5074:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:29
Start Date: 06/Aug/18 20:29
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6145: [BEAM-5074][SQL] 
improve BeamSqlRowComparator
URL: https://github.com/apache/beam/pull/6145#issuecomment-410842956
 
 
   We discussed this. You are going to post the code that requires this change 
and we can discuss this in the context of that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 131570)
Time Spent: 50m  (was: 40m)

> Make BeamSqlRowComparator more general
> --
>
> Key: BEAM-5074
> URL: https://issues.apache.org/jira/browse/BEAM-5074
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> BeamSqlRowComparator returns reversed comparison results (if a < b in terms 
> of value, BeamSqlRowComparator returns a > b). 
>  
> Add a flag to turn on/off this reversed effect. The reason of doing this is 
> to help implement ORDER BY ASC/DESC in the normal way.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6139 from amaliujia/rui_wang-remove_RelCollationTraitDef

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6429412228d91f0f03340e59cb01508b7fe70ecf
Merge: 943dc60 5ab3ed8
Author: Andrew Pilloud 
AuthorDate: Mon Aug 6 13:28:50 2018 -0700

Merge pull request #6139 from amaliujia/rui_wang-remove_RelCollationTraitDef

[BEAM-5072][SQL]Remove RelCollationTraitDef configuration in 
BeamQueryPlanner

 .../beam/sdk/extensions/sql/impl/BeamQueryPlanner.java |  4 +---
 .../apache/beam/sdk/extensions/sql/impl/JdbcDriver.java| 14 ++
 .../beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java |  4 ++--
 .../sql/impl/interpreter/BeamSqlFnExecutorTestBase.java|  4 +---
 4 files changed, 18 insertions(+), 8 deletions(-)

[jira] [Work logged] (BEAM-5072) SortRelNode is wrongly removed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5072?focusedWorklogId=131569&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-131569
 ]

ASF GitHub Bot logged work on BEAM-5072:


Author: ASF GitHub Bot
Created on: 06/Aug/18 20:28
Start Date: 06/Aug/18 20:28
Worklog Time Spent: 10m 
  Work Description: apilloud closed pull request #6139: 
[BEAM-5072][SQL]Remove RelCollationTraitDef configuration in BeamQueryPlanner
URL: https://github.com/apache/beam/pull/6139
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java
index 3d11ebf43a4..bdee34a7a96 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java
@@ -31,7 +31,6 @@
 import org.apache.calcite.plan.RelTraitDef;
 import org.apache.calcite.plan.RelTraitSet;
 import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.rel.RelCollationTraitDef;
 import org.apache.calcite.rel.RelRoot;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.calcite.sql.SqlNode;
@@ -76,8 +75,7 @@
 final SchemaPlus schema = connection.getRootSchema();
 final SchemaPlus defaultSchema = JdbcDriver.getDefaultSchema(connection);
 
-final ImmutableList traitDefs =
-ImmutableList.of(ConventionTraitDef.INSTANCE, 
RelCollationTraitDef.INSTANCE);
+final ImmutableList traitDefs = 
ImmutableList.of(ConventionTraitDef.INSTANCE);
 
 final CalciteCatalogReader catalogReader =
 new CalciteCatalogReader(
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
index fe7d572f547..f3ce0016165 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
@@ -24,6 +24,8 @@
 import com.google.common.base.Function;
 import java.sql.Connection;
 import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
 import java.util.Map;
 import java.util.Properties;
 import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
@@ -42,7 +44,10 @@
 import org.apache.calcite.jdbc.Driver;
 import org.apache.calcite.plan.RelOptPlanner;
 import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelTraitDef;
+import org.apache.calcite.rel.RelCollationTraitDef;
 import org.apache.calcite.rel.rules.CalcRemoveRule;
+import org.apache.calcite.rel.rules.SortRemoveRule;
 import org.apache.calcite.runtime.Hook;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.calcite.tools.RuleSet;
@@ -92,6 +97,15 @@ public Void apply(RelOptPlanner planner) {
   }
 }
 planner.removeRule(CalcRemoveRule.INSTANCE);
+planner.removeRule(SortRemoveRule.INSTANCE);
+
+List relTraitDefs = new 
ArrayList<>(planner.getRelTraitDefs());
+planner.clearRelTraitDefs();
+for (RelTraitDef def : relTraitDefs) {
+  if (!(def instanceof RelCollationTraitDef)) {
+planner.addRelTraitDef(def);
+  }
+}
 return null;
   }
 });
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
index 6c13da8b1fe..24d72c1fd67 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
@@ -54,7 +54,6 @@
 import org.apache.calcite.rel.rules.ProjectToCalcRule;
 import org.apache.calcite.rel.rules.PruneEmptyRules;
 import org.apache.calcite.rel.rules.SortProjectTransposeRule;
-import org.apache.calcite.rel.rules.SortRemoveRule;
 import org.apache.calcite.rel.rules.UnionEliminatorRule;
 import org.apache.calcite.rel.rules.UnionToDistinctRule;
 import org.apache.calcite.tools.RuleSet;
@@ -120,7 +119,8 @@
   // AggregateReduceFunctionsRule.INSTANCE,
 
   // remove unnecessary sort rule
-  SortRemoveRule.INSTANCE,
+ 

[beam] branch master updated (943dc60 -> 6429412)

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 943dc60  Fix beam_PerformanceTests_Python (#6142)
 add 5ab3ed8  remove RelCollationTraitDef
 new 6429412  Merge pull request #6139 from 
amaliujia/rui_wang-remove_RelCollationTraitDef

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/extensions/sql/impl/BeamQueryPlanner.java |  4 +---
 .../apache/beam/sdk/extensions/sql/impl/JdbcDriver.java| 14 ++
 .../beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java |  4 ++--
 .../sql/impl/interpreter/BeamSqlFnExecutorTestBase.java|  4 +---
 4 files changed, 18 insertions(+), 8 deletions(-)

[jira] [Created] (BEAM-5091) Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test suite.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5091:
-

 Summary: Exercise Python 3 SDK harness container in 
ValidatesContainer Jenkins test suite.
 Key: BEAM-5091
 URL: https://issues.apache.org/jira/browse/BEAM-5091
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)