Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #235

[Apache Jenkins Server]

See 


Changes:

[kenn] Remove Schema.TypeName.type()

[kenn] Remove unsafe builder methods from FieldType

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d506e1ef8968bf97432b522bf8589db4f1be29f6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d506e1ef8968bf97432b522bf8589db4f1be29f6
Commit message: "Merge pull request #5498: [BEAM-4076] Remove unsafe methods 
from Schema.TypeName and Schema.FieldType"
 > git rev-list --no-walk 123c20e9ad754cc1142a0aed17edbbef637b6176 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2087248872860007196.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins411778276205158.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4125654345617137736.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-235
Error from server (AlreadyExists): namespaces "filebasedioithdfs-235" already 
exists
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #332

[Apache Jenkins Server]

See 


Changes:

[kenn] Remove Schema.TypeName.type()

[kenn] Remove unsafe builder methods from FieldType

--
[...truncated 118.76 KB...]
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)

May 31, 2018 6:05:58 AM org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.createTable(DatabaseTestHelper.java:46)
at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT.setUp(HadoopInputFormatIOIT.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #241

[Apache Jenkins Server]

See 


Changes:

[kenn] Remove Schema.TypeName.type()

[kenn] Remove unsafe builder methods from FieldType

--
[...truncated 203.78 KB...]
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.192.159.173:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.192.159.173:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #607

[Apache Jenkins Server]

See 


--
[...truncated 18.31 MB...]
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:08.823Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/ParDo(GarbageCollectTemporaryFiles)
 into BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/Values/Values/Map
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:08.849Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify into 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:08.883Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/GroupByKey/GroupByWindow 
into BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Read
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.024Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write into 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.059Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
BigQueryIO.Write/BatchLoads/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.086Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
BigQueryIO.Write/BatchLoads/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.113Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/ReifyRenameInput/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
 into 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.143Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables) 
into BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/ExpandIterable
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.177Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Reify into 
BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/Window.Into()/Window.Assign
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.207Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow 
into BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.239Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Write into 
BigQueryIO.Write/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Reify
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.266Z: Unzipping flatten s46 for input 
s43.writtenFiles
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.300Z: Fusing unzipped copy of 
BigQueryIO.Write/BatchLoads/ReifyResults/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow),
 through flatten BigQueryIO.Write/BatchLoads/FlattenFiles, into producer 
BigQueryIO.Write/BatchLoads/WriteBundlesToFiles
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.333Z: Fusing consumer 
BigQueryIO.Write/BatchLoads/ReifyResults/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
 into BigQueryIO.Write/BatchLoads/WriteGroupedRecords
May 31, 2018 6:44:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T06:44:09.368Z: Fusing consumer 

[jira] [Work logged] (BEAM-4356) Enforce ErrorProne analysis in the nexmark project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4356?focusedWorklogId=107592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107592
 ]

ASF GitHub Bot logged work on BEAM-4356:


Author: ASF GitHub Bot
Created on: 31/May/18 07:44
Start Date: 31/May/18 07:44
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5513: 
[BEAM-4356] Enforce ErrorProne analysis in nexmark
URL: https://github.com/apache/beam/pull/5513#discussion_r192013297
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java
 ##
 @@ -25,6 +25,7 @@
 /**
  * A set of {@link NexmarkConfiguration}s.
  */
+@SuppressWarnings("ImmutableEnumChecker")
 
 Review comment:
   I agree that the idea of fixing errorprone issues is not to just mark things 
to be ignored, but here there is a bigger issue.  This change is not so simple, 
at the first try I just changed the Lists to be immutable, but errorprone 
requires the type of the object in the list [to be immutable 
too](http://errorprone.info/bugpattern/ImmutableEnumChecker).
   
   So fixing this implies making `NexmarkConfiguration` an Immutable object and 
this has a considerable impact so I prefer that this is covered in a different 
ticket (also to address the repeated options with NexmarkOptions). I just 
created [BEAM-4433](https://issues.apache.org/jira/browse/BEAM-4433) for this. 
Feel free to take it @cademarkegard if you are interested on working on it.
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107592)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in the nexmark project
> --
>
> Key: BEAM-4356
> URL: https://issues.apache.org/jira/browse/BEAM-4356
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-nexmark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-nexmark:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/nexmark}} project.
> # In {{sdks/java/nexmark/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4323.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Enforce ErrorProne analysis in sketching extensions project
> ---
>
> Key: BEAM-4323
> URL: https://issues.apache.org/jira/browse/BEAM-4323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-sketching
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
> project.
> # In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107597
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 07:53
Start Date: 31/May/18 07:53
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5521: [BEAM-4303] Enforce 
ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#issuecomment-393443297
 
 
   R: @swegner @iemejia
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107597)
Time Spent: 20m  (was: 10m)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107600
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 08:02
Start Date: 31/May/18 08:02
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5521: [BEAM-4303] Enforce 
ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#issuecomment-393445779
 
 
   @swegner is there a way to mark ignores at the module level ? 
   I think we should use that only in this case 
[StringSplitter](http://errorprone.info/bugpattern/StringSplitter) 
   
   We should not expose guava's Splitter in the most basic examples, e.g. 
Wordcount/WindowedWordcount (because those are commonly copy pasted) and using 
the `@SuppressWarnings("StringSplitter")` also does not seem natural for a 
baginner example too so better do it at the module level.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107600)
Time Spent: 0.5h  (was: 20m)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4257) Add error reason and table destination to BigQueryIO streaming failed inserts

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4257?focusedWorklogId=107604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107604
 ]

ASF GitHub Bot logged work on BEAM-4257:


Author: ASF GitHub Bot
Created on: 31/May/18 08:18
Start Date: 31/May/18 08:18
Worklog Time Spent: 10m 
  Work Description: calonso commented on issue #5341: [BEAM-4257] Increases 
BigQuery streaming error information
URL: https://github.com/apache/beam/pull/5341#issuecomment-393451323
 
 
   Hi @reuvenlax, just wanted to friendly remind you about this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107604)
Time Spent: 2h  (was: 1h 50m)

> Add error reason and table destination to BigQueryIO streaming failed inserts
> -
>
> Key: BEAM-4257
> URL: https://issues.apache.org/jira/browse/BEAM-4257
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Carlos Alonso
>Assignee: Carlos Alonso
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> When using `BigQueryIO.Write` and getting `WriteResult.getFailedInserts()` we 
> get a `PCollection` which is fine, but in order to properly work on 
> the errors downstream having extended information such as the `InsertError` 
> fields and the `TableReference` it was routed to would be really valuable.
>  
> My suggestion is to create a new object that contains all that information 
> and return a `PCollection` of those instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4356) Enforce ErrorProne analysis in the nexmark project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4356?focusedWorklogId=107612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107612
 ]

ASF GitHub Bot logged work on BEAM-4356:


Author: ASF GitHub Bot
Created on: 31/May/18 08:42
Start Date: 31/May/18 08:42
Worklog Time Spent: 10m 
  Work Description: echauchot closed pull request #5513: [BEAM-4356] 
Enforce ErrorProne analysis in nexmark
URL: https://github.com/apache/beam/pull/5513
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/nexmark/build.gradle b/sdks/java/nexmark/build.gradle
index 53739af1ac5..bc7a0f0e4b3 100644
--- a/sdks/java/nexmark/build.gradle
+++ b/sdks/java/nexmark/build.gradle
@@ -17,7 +17,7 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature()
+applyJavaNature(failOnWarning: true)
 
 description = "Apache Beam :: SDKs :: Java :: Nexmark"
 
@@ -43,6 +43,7 @@ configurations {
 
 dependencies {
   compile library.java.guava
+  compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")
   shadow project(path: 
":beam-sdks-java-extensions-google-cloud-platform-core", configuration: 
"shadow")
@@ -63,6 +64,7 @@ dependencies {
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   shadow library.java.slf4j_jdk14
   testCompile library.java.hamcrest_core
+  testCompileOnly library.java.findbugs_annotations
 
   gradleRun project(path: project.path, configuration: "shadow")
   gradleRun project(path: nexmarkRunnerDependency, configuration: "shadow")
diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java
 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java
index 14a91bd71a3..592d1aa9c06 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java
@@ -172,19 +172,16 @@
   void setUseWallclockEventTime(Boolean useWallclockEventTime);
 
   @Description("Assert pipeline results match model results.")
-  @Nullable
   boolean getAssertCorrectness();
 
   void setAssertCorrectness(boolean assertCorrectness);
 
   @Description("Log all input events.")
-  @Nullable
   boolean getLogEvents();
 
   void setLogEvents(boolean logEvents);
 
   @Description("Log all query results.")
-  @Nullable
   boolean getLogResults();
 
   void setLogResults(boolean logResults);
@@ -282,7 +279,6 @@
 
   @Description("If true, don't run the actual query. Instead, calculate the 
distribution "
+ "of number of query results per (event time) minute according 
to the query model.")
-  @Nullable
   boolean getJustModelResultRate();
 
   void setJustModelResultRate(boolean justModelResultRate);
diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java
index a6e9f6b2a20..3a6e1174d36 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java
@@ -25,6 +25,7 @@
 /**
  * A set of {@link NexmarkConfiguration}s.
  */
+@SuppressWarnings("ImmutableEnumChecker")
 public enum NexmarkSuite {
   /**
* The default.
diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
index 7a1d1f1c2b8..9f40eb85c2b 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
@@ -57,7 +57,7 @@ public int hashCode() {
 AUCTION(1),
 BID(2);
 
-private int value = -1;
+private final int value;
 
 Tag(int value) {
   this.value = value;
diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
index 24d4d044f4e..a5f834bf682 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
@@ -143,6 +143,8 @@ private String timingToString(PaneInfo.Timing timing) {
 return "O";
   case LATE:
 return "L";
+  case UNKNOWN:
+return "U";
 }
 throw new RuntimeException(); // cases are exhaustive
   

[beam] branch master updated (de18926 -> a0d1ad1)

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from de18926  Merge pull request #5501: [BEAM-4323] Enforce ErrorProne 
analysis in sketching extensions
 add 51138e4  [BEAM-4356] Enforce ErrorProne analysis in nexmark
 new a0d1ad1  Merge pull request #5513 from 
iemejia/BEAM-4356-errorprone-nexmark

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/nexmark/build.gradle| 4 +++-
 .../src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java | 4 
 .../src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java   | 1 +
 .../src/main/java/org/apache/beam/sdk/nexmark/model/Event.java| 2 +-
 .../src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java| 2 ++
 .../beam/sdk/nexmark/sources/generator/model/AuctionGenerator.java| 2 +-
 6 files changed, 8 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
echauc...@apache.org.

[beam] 01/01: Merge pull request #5513 from iemejia/BEAM-4356-errorprone-nexmark

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9
Merge: de18926 51138e4
Author: Etienne Chauchot 
AuthorDate: Thu May 31 10:42:39 2018 +0200

Merge pull request #5513 from iemejia/BEAM-4356-errorprone-nexmark

Merge pull request #5513: [BEAM-4356] Enforce ErrorProne analysis in nexmark

 sdks/java/nexmark/build.gradle| 4 +++-
 .../src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java | 4 
 .../src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java   | 1 +
 .../src/main/java/org/apache/beam/sdk/nexmark/model/Event.java| 2 +-
 .../src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java| 2 ++
 .../beam/sdk/nexmark/sources/generator/model/AuctionGenerator.java| 2 +-
 6 files changed, 8 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
echauc...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #236

[Apache Jenkins Server]

See 


Changes:

[kenn] Remove Schema.TypeName.type()

[kenn] Remove unsafe builder methods from FieldType

--
[...truncated 371.30 KB...]
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn.processElement(FileBasedIOITHelper.java:78)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
/.temp-beam-2018-05-31_06-05-13-0/ca09e082-ad1c-4320-bafe-e31297083327 could 
only be replicated to 0 nodes instead of minReplication (=1).  There are 0 
datanode(s) running and no node(s) are excluded in this operation.
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
   

[jira] [Created] (BEAM-4433) Nexmark Configuration should be an immutable object and clean from repeated options

[JIRA]

Ismaël Mejía created BEAM-4433:
--

 Summary: Nexmark Configuration should be an immutable object and 
clean from repeated options
 Key: BEAM-4433
 URL: https://issues.apache.org/jira/browse/BEAM-4433
 Project: Beam
  Issue Type: Improvement
  Components: examples-nexmark
Reporter: Ismaël Mejía
Assignee: Kenneth Knowles


{{NexmarkConfiguration}} object is not immutable, and is arbitrarily changed in 
different parts making tracking of state changes hard and error prone.

Also it has some repeated configuration options that exist in 
{{NexmarkOptions}} too. Those should be removed and let only in one place, 
ideally NexmarkOptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4433) Nexmark Configuration should be an immutable object and clean from repeated options

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4433:
--

Assignee: (was: Kenneth Knowles)

> Nexmark Configuration should be an immutable object and clean from repeated 
> options
> ---
>
> Key: BEAM-4433
> URL: https://issues.apache.org/jira/browse/BEAM-4433
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Ismaël Mejía
>Priority: Minor
>
> {{NexmarkConfiguration}} object is not immutable, and is arbitrarily changed 
> in different parts making tracking of state changes hard and error prone.
> Also it has some repeated configuration options that exist in 
> {{NexmarkOptions}} too. Those should be removed and let only in one place, 
> ideally NexmarkOptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4394) Consider enabling spotless java format throughout codebase

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496259#comment-16496259
 ] 

Ismaël Mejía commented on BEAM-4394:


I agree with good communication the impact is low and this is what is not 
happening in this issue (and in the PR). 

There are some issues:

- Other devs can disagree (which I doubt given the good arguments) but better 
discuss than force things.
- It could break all other ongoing PRs (causing frustration to contributors).
- Contributors must be aware of this autoformat enforcing, this implies 
updating the contribution guide + communicating in the ML.

About how to do it I don’t have a strict preference, I think the impact the 
lowest impact is if it is done all at once. If you guys have someone who can do 
it all at once e.g. in a day with low PR traffic, it is probably better, but 
all people in the community deserve to be aware of this given that it impacts 
their work.

> Consider enabling spotless java format throughout codebase
> --
>
> Key: BEAM-4394
> URL: https://issues.apache.org/jira/browse/BEAM-4394
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Kenneth Knowles
>Priority: Major
>
> "Spotless" can enforce - and automatically restore - automatic Java 
> formatting. Whenever formatting is off, it tells a user the exact command to 
> fix it.
> It isn't (just) about code layout, it is about automation. We have pretty 
> strict style rules enforced by checkstyle. The most efficient way to fix up a 
> file is with autoformat. But if the autoformat hits a bunch of irrelevant 
> lines, that is annoying for a reviewer and obscures git blame.
> If we enforce autoformat all the time, then it makes sure that autoformatting 
> a particular PR has minimal effects and is always safe to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4356) Enforce ErrorProne analysis in the nexmark project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4356?focusedWorklogId=107608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107608
 ]

ASF GitHub Bot logged work on BEAM-4356:


Author: ASF GitHub Bot
Created on: 31/May/18 08:40
Start Date: 31/May/18 08:40
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5513: 
[BEAM-4356] Enforce ErrorProne analysis in nexmark
URL: https://github.com/apache/beam/pull/5513#discussion_r192026914
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkSuite.java
 ##
 @@ -25,6 +25,7 @@
 /**
  * A set of {@link NexmarkConfiguration}s.
  */
+@SuppressWarnings("ImmutableEnumChecker")
 
 Review comment:
   Fair enough, let's tackle it in another ticket, LGTM merging


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107608)
Time Spent: 1h  (was: 50m)

> Enforce ErrorProne analysis in the nexmark project
> --
>
> Key: BEAM-4356
> URL: https://issues.apache.org/jira/browse/BEAM-4356
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-nexmark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-nexmark:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/nexmark}} project.
> # In {{sdks/java/nexmark/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107609
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 08:40
Start Date: 31/May/18 08:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192022213
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
 ##
 @@ -146,13 +164,77 @@ private void appendPerf(
   private static final String LINE =
   
"==";
 
-  /**
-   * Print summary  of {@code actual} vs (if non-null) {@code baseline}.
-   */
+  /** Send {@code nexmarkPerf} to BigQuery. */
+  @VisibleForTesting
+  static void writeQueryPerftoBigQuery(
 
 Review comment:
   Can we perform this write using a Beam pipeline that uses DirectRunner and 
BigQuery sink instead of using BigQueryService class ? I think you should use 
either that or use a BigQuery client library directly instead of using the 
intermediate "BigQueryService" class in Beam which is not intended to be a 
public utility.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107609)
Time Spent: 1h  (was: 50m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107611
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 08:40
Start Date: 31/May/18 08:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192025438
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
 ##
 @@ -420,6 +428,9 @@ public String toShortString() {
 if (sinkType != DEFAULT.sinkType) {
   sb.append(String.format("; sinkType:%s", sinkType));
 }
+if (exportSummaryToBigQuery != DEFAULT.exportSummaryToBigQuery) {
 
 Review comment:
   Why are we comparing with default values ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107611)
Time Spent: 1h 20m  (was: 1h 10m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107610
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 08:40
Start Date: 31/May/18 08:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192024638
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
 ##
 @@ -47,6 +47,10 @@
   @JsonProperty
   public NexmarkUtils.SinkType sinkType = NexmarkUtils.SinkType.DEVNULL;
 
+  /** Shall we export the summary to BigQuery. */
 
 Review comment:
   How about "If true, exports the summary to BigQuery." ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107610)
Time Spent: 1h 10m  (was: 1h)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #384

[Apache Jenkins Server]

See 


Changes:

[kenn] Remove Schema.TypeName.type()

[kenn] Remove unsafe builder methods from FieldType

--
[...truncated 18.18 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 31, 2018 6:36:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 31, 2018 6:36:27 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531063621-fe0660b7/output/results/staging/
May 31, 2018 6:36:27 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash D5ZTy77eB1SxK4pEF55Epw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531063621-fe0660b7/output/results/staging/pipeline-D5ZTy77eB1SxK4pEF55Epw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 6:36:28 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-30_23_36_27-8374727479302907689?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-30_23_36_27-8374727479302907689

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 6:36:28 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-30_23_36_27-8374727479302907689
May 31, 2018 6:36:28 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-30_23_36_27-8374727479302907689 with 1 
expected assertions.
May 31, 2018 6:36:38 AM 

[beam] 01/01: Merge pull request #5501: [BEAM-4323] Enforce ErrorProne analysis in sketching extensions

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit de18926c61ff1ddb9b543d2df69cff94941d5259
Merge: d506e1e ea3a9b3
Author: Ismaël Mejía 
AuthorDate: Thu May 31 09:50:42 2018 +0200

Merge pull request #5501: [BEAM-4323] Enforce ErrorProne analysis in 
sketching extensions

 sdks/java/extensions/sketching/build.gradle|  4 +++-
 .../extensions/sketching/ApproximateDistinct.java  |  7 +++---
 .../extensions/sketching/SketchFrequencies.java| 26 ++--
 .../sdk/extensions/sketching/TDigestQuantiles.java |  2 +-
 .../sketching/ApproximateDistinctTest.java |  2 +-
 .../sketching/SketchFrequenciesTest.java   | 28 ++
 .../extensions/sketching/TDigestQuantilesTest.java | 10 
 7 files changed, 40 insertions(+), 39 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (d506e1e -> de18926)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d506e1e  Merge pull request #5498: [BEAM-4076] Remove unsafe methods 
from Schema.TypeName and Schema.FieldType
 add ea3a9b3  [BEAM-4323] Enforce ErrorProne analysis in sketching 
extensions
 new de18926  Merge pull request #5501: [BEAM-4323] Enforce ErrorProne 
analysis in sketching extensions

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sketching/build.gradle|  4 +++-
 .../extensions/sketching/ApproximateDistinct.java  |  7 +++---
 .../extensions/sketching/SketchFrequencies.java| 26 ++--
 .../sdk/extensions/sketching/TDigestQuantiles.java |  2 +-
 .../sketching/ApproximateDistinctTest.java |  2 +-
 .../sketching/SketchFrequenciesTest.java   | 28 ++
 .../extensions/sketching/TDigestQuantilesTest.java | 10 
 7 files changed, 40 insertions(+), 39 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Work logged] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4323?focusedWorklogId=107595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107595
 ]

ASF GitHub Bot logged work on BEAM-4323:


Author: ASF GitHub Bot
Created on: 31/May/18 07:50
Start Date: 31/May/18 07:50
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #5501: [BEAM-4323] Enforce 
ErrorProne analysis in sketching extensions
URL: https://github.com/apache/beam/pull/5501
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/extensions/sketching/build.gradle 
b/sdks/java/extensions/sketching/build.gradle
index 1ddc2269ab5..6392b0842ac 100644
--- a/sdks/java/extensions/sketching/build.gradle
+++ b/sdks/java/extensions/sketching/build.gradle
@@ -17,7 +17,7 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature()
+applyJavaNature(failOnWarning: true)
 
 description = "Apache Beam :: SDKs :: Java :: Extensions :: Sketching"
 
@@ -26,10 +26,12 @@ def tdigest_version = "3.2"
 
 dependencies {
   compile library.java.guava
+  compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow "com.clearspring.analytics:stream:$streamlib_version"
   shadow "com.tdunning:t-digest:$tdigest_version"
   shadow library.java.slf4j_api
+  testCompileOnly library.java.findbugs_annotations
   shadowTest library.java.avro
   shadowTest library.java.commons_lang3
   shadowTest project(path: ":beam-sdks-java-core", configuration: "shadowTest")
diff --git 
a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
 
b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
index e8d85f7c057..2d0a56ee4d4 100644
--- 
a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
+++ 
b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
@@ -534,9 +534,8 @@ protected long getEncodedElementByteSize(HyperLogLogPlus 
value) throws IOExcepti
* Utility class that provides {@link DoFn}s to retrieve the cardinality 
from a {@link
* HyperLogLogPlus} structure in a global or perKey context.
*/
-  public static class RetrieveCardinality {
-
-public static  DoFn, KV> perKey() {
+  private static class RetrieveCardinality {
+private static  DoFn, KV> perKey() {
   return new DoFn, KV>() {
 @ProcessElement
 public void processElement(ProcessContext c) {
@@ -546,7 +545,7 @@ public void processElement(ProcessContext c) {
   };
 }
 
-public static DoFn globally() {
+private static DoFn globally() {
   return new DoFn() {
 @ProcessElement
 public void apply(ProcessContext c) {
diff --git 
a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
 
b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
index 5d06ec00ac5..0451dcebd23 100644
--- 
a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
+++ 
b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
@@ -148,7 +148,7 @@
  * {@literal PCollection>} pairs = pc.apply(ParDo.of(
  *{@literal new DoFn>()} {
  *  {@literal @ProcessElement}
- *   public void procesElement(ProcessContext c) {
+ *   public void processElement(ProcessContext c) {
  * Long elem = c.element();
  * CountMinSketch sketch = c.sideInput(sketchView);
  * c.output(sketch.estimateCount(elem, coder));
@@ -260,10 +260,11 @@
 
 @Override
 public PCollection> expand(PCollection input) {
-  return input.apply("Compute Count-Min Sketch",
-  Combine.>globally(CountMinSketchFn
-  .create(input.getCoder())
-  .withAccuracy(relativeError(), confidence(;
+  return input.apply(
+  "Compute Count-Min Sketch",
+  Combine.globally(
+  CountMinSketchFn.create(input.getCoder())
+  .withAccuracy(relativeError(), confidence(;
 }
   }
 
@@ -324,10 +325,11 @@
 @Override
 public PCollection>> expand(PCollection> input) {
   KvCoder inputCoder = (KvCoder) input.getCoder();
-  return input.apply("Compute Count-Min Sketch perKey",
-  Combine.>perKey(CountMinSketchFn
-  

[jira] [Work logged] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4323?focusedWorklogId=107596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107596
 ]

ASF GitHub Bot logged work on BEAM-4323:


Author: ASF GitHub Bot
Created on: 31/May/18 07:51
Start Date: 31/May/18 07:51
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5501: [BEAM-4323] Enforce 
ErrorProne analysis in sketching extensions
URL: https://github.com/apache/beam/pull/5501#issuecomment-393442598
 
 
   Thanks @chamikaramj !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107596)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in sketching extensions project
> ---
>
> Key: BEAM-4323
> URL: https://issues.apache.org/jira/browse/BEAM-4323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-sketching
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
> project.
> # In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4418) Improve gradle integration with IntelliJ

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496234#comment-16496234
 ] 

Etienne Chauchot commented on BEAM-4418:


I can confirm what you are saying Kenn: failOnWarnings occur for deprecations 
for me also and also with the delegating build to gradle.

A comment on your second comment: strange indeed considering that gradle should 
use only artifacts built from the current project and not the ones in a repo 
(like maven)

> Improve gradle integration with IntelliJ
> 
>
> Key: BEAM-4418
> URL: https://issues.apache.org/jira/browse/BEAM-4418
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Etienne Chauchot
>Priority: Major
>
> To be able to work efficiently with gradle, the integration with intelliJ 
> (more common IDE in the community I think) needs to be improved.The aim of 
> this ticket is to gather areas of improvement discovered by people. Feel free 
> to comment on what you discovered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #608

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4323] Enforce ErrorProne analysis in sketching extensions

--
[...truncated 17.85 MB...]
May 31, 2018 8:33:15 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_01_33_13-5924242805326422357
May 31, 2018 8:33:15 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_01_33_13-5924242805326422357 with 0 
expected assertions.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:14.095Z: Autoscaling is enabled for job 
2018-05-31_01_33_13-5924242805326422357. The number of workers will be between 
1 and 1000.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:14.142Z: Autoscaling was automatically enabled for 
job 2018-05-31_01_33_13-5924242805326422357.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:16.807Z: Checking required Cloud APIs are enabled.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:16.985Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:20.503Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.043Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.279Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.316Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.571Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.605Z: Elided trivial flatten 
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.648Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.685Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.729Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.771Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.811Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 31, 2018 8:33:29 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:33:21.857Z: Fusing consumer 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #385

[Apache Jenkins Server]

See 


--
[...truncated 18.22 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 31, 2018 8:48:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 31, 2018 8:48:34 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531084830-486a081/output/results/staging/
May 31, 2018 8:48:35 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash Uy5OfkIByk5KVpTtPk77Lw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531084830-486a081/output/results/staging/pipeline-Uy5OfkIByk5KVpTtPk77Lw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 8:48:36 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_01_48_35-1399331152337114576?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-31_01_48_35-1399331152337114576

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 8:48:36 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_01_48_35-1399331152337114576
May 31, 2018 8:48:36 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_01_48_35-1399331152337114576 with 1 
expected assertions.
May 31, 2018 8:48:49 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T08:48:35.517Z: Autoscaling is enabled for job 
2018-05-31_01_48_35-1399331152337114576. The number of workers will be between 
1 and 1000.
May 31, 2018 8:48:49 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler 

[beam-site] branch beam2726 deleted (was 66eaf09)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch beam2726
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 was 66eaf09  [BEAM-2726] Fix typo in programming guide

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam-site] branch snapshot-release deleted (was 9464aed)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch snapshot-release
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 was 9464aed  Add nightly snapshot build to the release guide

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #333

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #236

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4323] Enforce ErrorProne analysis in sketching extensions

[iemejia] [BEAM-4356] Enforce ErrorProne analysis in nexmark

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9
Commit message: "Merge pull request #5513 from 
iemejia/BEAM-4356-errorprone-nexmark"
 > git rev-list --no-walk d506e1ef8968bf97432b522bf8589db4f1be29f6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3726649809820674123.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5272130740108952383.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins878509607385604741.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-236
Error from server (AlreadyExists): namespaces "filebasedioithdfs-236" already 
exists
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #236

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4323] Enforce ErrorProne analysis in sketching extensions

[iemejia] [BEAM-4356] Enforce ErrorProne analysis in nexmark

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9
Commit message: "Merge pull request #5513 from 
iemejia/BEAM-4356-errorprone-nexmark"
 > git rev-list --no-walk d506e1ef8968bf97432b522bf8589db4f1be29f6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8332475114072752159.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6433075283205592077.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1064739421730956114.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-236
Error from server (AlreadyExists): namespaces "filebasedioithdfs-236" already 
exists
Build step 'Execute shell' marked build as failure

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #237

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_MongoDBIO_IT #242

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107622
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 09:42
Start Date: 31/May/18 09:42
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5464: 
[BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192043662
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
 ##
 @@ -146,13 +164,77 @@ private void appendPerf(
   private static final String LINE =
   
"==";
 
-  /**
-   * Print summary  of {@code actual} vs (if non-null) {@code baseline}.
-   */
+  /** Send {@code nexmarkPerf} to BigQuery. */
+  @VisibleForTesting
+  static void writeQueryPerftoBigQuery(
 
 Review comment:
   For your 2 questions:
   1. No, here we output to BigQuery only the response time of the pipeline so 
it needs to be finished. 
   2. What you suggest (bigquery client) is what I did at first and I asked you 
how I can unit test it and you answered that I should use FakeBigQueryServices. 
But to use FakeBigQueryServices I needed to refactor everything to use 
FakeBigQueryServices's super class (BigQueryServices)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107622)
Time Spent: 1.5h  (was: 1h 20m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #386

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4323] Enforce ErrorProne analysis in sketching extensions

[iemejia] [BEAM-4356] Enforce ErrorProne analysis in nexmark

--
[...truncated 18.23 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 31, 2018 11:05:13 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 31, 2018 11:05:13 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531110509-daf4e307/output/results/staging/
May 31, 2018 11:05:13 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash VCCegX3cP2HMMINVQemE5g> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531110509-daf4e307/output/results/staging/pipeline-VCCegX3cP2HMMINVQemE5g.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 11:05:15 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_04_05_14-10093640796875245231?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-31_04_05_14-10093640796875245231

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 11:05:15 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_04_05_14-10093640796875245231
May 31, 2018 11:05:15 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-31_04_05_14-10093640796875245231 with 1 
expected 

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107627
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 09:51
Start Date: 31/May/18 09:51
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #5464: [BEAM-4283] Write 
Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#issuecomment-393478182
 
 
   @chamikaramj Thanks for your review, I answered all your comments. PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107627)
Time Spent: 2h  (was: 1h 50m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4356) Enforce ErrorProne analysis in the nexmark project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4356.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Enforce ErrorProne analysis in the nexmark project
> --
>
> Key: BEAM-4356
> URL: https://issues.apache.org/jira/browse/BEAM-4356
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-nexmark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-nexmark:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/nexmark}} project.
> # In {{sdks/java/nexmark/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #609

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4356] Enforce ErrorProne analysis in nexmark

--
[...truncated 18.01 MB...]
INFO: Uploading <115879 bytes, hash 2jWDI-tLwpinuztpmhMCew> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0531092404-9a1d8b7/output/results/staging/pipeline-2jWDI-tLwpinuztpmhMCew.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 9:24:16 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_02_24_15-995925806846606392?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-31_02_24_15-995925806846606392

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 9:24:16 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_02_24_15-995925806846606392
May 31, 2018 9:24:16 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_02_24_15-995925806846606392 with 0 
expected assertions.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:15.536Z: Autoscaling is enabled for job 
2018-05-31_02_24_15-995925806846606392. The number of workers will be between 1 
and 1000.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:15.581Z: Autoscaling was automatically enabled for 
job 2018-05-31_02_24_15-995925806846606392.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:19.706Z: Checking required Cloud APIs are enabled.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:19.907Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:23.623Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.047Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.299Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.348Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.634Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.668Z: Elided trivial flatten 
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.715Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.764Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T09:24:24.811Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 9:24:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler 

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107625
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 09:46
Start Date: 31/May/18 09:46
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5464: 
[BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192044761
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
 ##
 @@ -47,6 +47,10 @@
   @JsonProperty
   public NexmarkUtils.SinkType sinkType = NexmarkUtils.SinkType.DEVNULL;
 
+  /** Shall we export the summary to BigQuery. */
 
 Review comment:
   If false, the summary is only output to the console.
   If true the summary is output to the console and it's content is written to 
bigquery tables per query x runner x mode.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107625)
Time Spent: 1h 40m  (was: 1.5h)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=107626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107626
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 31/May/18 09:50
Start Date: 31/May/18 09:50
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5464: 
[BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r192045764
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
 ##
 @@ -420,6 +428,9 @@ public String toShortString() {
 if (sinkType != DEFAULT.sinkType) {
   sb.append(String.format("; sinkType:%s", sinkType));
 }
+if (exportSummaryToBigQuery != DEFAULT.exportSummaryToBigQuery) {
 
 Review comment:
   Like all the other configuration items,  when we print the configuration, we 
print their value only if they are set by the user (i.e. their value is not the 
default value)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107626)
Time Spent: 1h 50m  (was: 1h 40m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107741
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 16:27
Start Date: 31/May/18 16:27
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5521: [BEAM-4303] Enforce 
ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#issuecomment-393591276
 
 
   @iemejia currently we have a [global set of 
excludes](https://github.com/apache/beam/blob/f6dd249559b262e598be12c8df19bcfb4aa133bd/build_rules.gradle#L426)
 in the `build_rules.gradle` `applyJavaNature()` block. We could add a 
parameter to the 
[`JavaNatureConfiguration`](https://github.com/apache/beam/blob/f6dd249559b262e598be12c8df19bcfb4aa133bd/build_rules.gradle#L339)
 such that modules can add their own, but my preference is to avoid that until 
we really need it.
   
   I agree with making examples simple and avoiding a Guava dependency, but 
also note that examples tend to get copy-pasted many times, so we should try to 
avoid known anti-patterns if possible. As @cademarkegard points out, the 
[StringSplitter ErrorProne 
page](http://errorprone.info/bugpattern/StringSplitter) also suggests using 
`String.split(String, int)` with a `limit` of `-1`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107741)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4249) beam_PostCommit_Python_ValidatesContainer_Dataflow red for a few days

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496830#comment-16496830
 ] 

Alan Myrvold commented on BEAM-4249:


Fixed by renaming the job.

 

https://builds.apache.org/job/beam_PostCommit_Py_ValCont/

> beam_PostCommit_Python_ValidatesContainer_Dataflow red for a few days
> -
>
> Key: BEAM-4249
> URL: https://issues.apache.org/jira/browse/BEAM-4249
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/
> Installing collected packages: protobuf, futures, grpcio, grpcio-tools
> Could not install packages due to an EnvironmentError: [Errno 13] Permission 
> denied: 
> '/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth'
> Consider using the `--user` option or check the permissions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4249) beam_PostCommit_Python_ValidatesContainer_Dataflow red for a few days

[Alan Myrvold (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Myrvold resolved BEAM-4249.

   Resolution: Fixed
Fix Version/s: 2.5.0

> beam_PostCommit_Python_ValidatesContainer_Dataflow red for a few days
> -
>
> Key: BEAM-4249
> URL: https://issues.apache.org/jira/browse/BEAM-4249
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/
> Installing collected packages: protobuf, futures, grpcio, grpcio-tools
> Could not install packages due to an EnvironmentError: [Errno 13] Permission 
> denied: 
> '/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth'
> Consider using the `--user` option or check the permissions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (f6dd249 -> 1414dd5)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f6dd249  Merge pull request #5506: [BEAM-3773] Build JDBC jar
 add f620e9c  Avoid fully qualified class names, when there is an existing 
(static) import
 new 1414dd5  Merge pull request #4748: Avoid fully qualified class names, 
when there is an existing (static)…

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/runners/apex/ApexYarnLauncher.java |  2 +-
 .../beam/runners/apex/ApexYarnLauncherTest.java|  4 +-
 .../core/construction/CoderTranslationTest.java|  2 +-
 .../construction/PCollectionTranslationTest.java   |  9 ++--
 .../core/construction/ParDoTranslationTest.java| 13 +++---
 .../core/construction/ReadTranslationTest.java |  5 +--
 .../core/construction/ReplacementOutputsTest.java  |  4 +-
 .../core/construction/SdkComponentsTest.java   |  3 +-
 .../construction/TestStreamTranslationTest.java|  3 +-
 .../WindowingStrategyTranslationTest.java  |  3 +-
 .../beam/runners/core/ReduceFnRunnerTest.java  | 48 +++---
 .../SimplePushbackSideInputDoFnRunnerTest.java |  5 +--
 .../beam/runners/core/StateInternalsTest.java  |  2 +-
 .../direct/UnboundedReadEvaluatorFactory.java  |  3 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java|  6 +--
 .../runners/direct/CloningBundleFactoryTest.java   |  3 +-
 .../runners/direct/DirectGraphVisitorTest.java |  2 +-
 .../direct/DirectTransformExecutorTest.java|  8 ++--
 ...cycleManagerRemovingTransformEvaluatorTest.java |  2 +-
 .../beam/runners/direct/EvaluationContextTest.java |  5 +--
 .../direct/ImmutableListBundleFactoryTest.java |  9 ++--
 .../beam/runners/direct/ParDoEvaluatorTest.java|  2 +-
 .../runners/direct/StepTransformResultTest.java|  3 +-
 .../direct/UnboundedReadEvaluatorFactoryTest.java  |  4 +-
 .../runners/direct/ViewOverrideFactoryTest.java|  5 +--
 .../beam/runners/direct/WatermarkManagerTest.java  | 17 
 .../flink/FlinkStreamingTransformTranslators.java  |  2 +-
 .../gearpump/translators/GroupByKeyTranslator.java |  4 +-
 .../beam/runners/dataflow/BatchViewOverrides.java  |  3 +-
 .../beam/runners/dataflow/DataflowMetrics.java | 32 +++
 .../beam/runners/dataflow/DataflowRunner.java  |  3 +-
 .../dataflow/PrimitiveParDoSingleFactory.java  |  2 +-
 .../beam/runners/dataflow/util/MonitoringUtil.java |  2 +-
 .../dataflow/DataflowPipelineTranslatorTest.java   | 10 ++---
 .../beam/runners/dataflow/DataflowRunnerTest.java  |  2 +-
 .../beam/runners/spark/io/SourceDStream.java   |  2 +-
 .../apache/beam/runners/spark/io/SourceRDD.java|  8 ++--
 .../runners/spark/stateful/StateSpecFunctions.java |  4 +-
 .../java/org/apache/beam/sdk/coders/AvroCoder.java |  6 +--
 .../main/java/org/apache/beam/sdk/io/FileIO.java   |  2 +-
 .../java/org/apache/beam/sdk/PipelineTest.java |  9 ++--
 .../sdk/options/PipelineOptionsReflectorTest.java  | 12 +++---
 .../beam/sdk/runners/TransformHierarchyTest.java   | 14 +++
 .../org/apache/beam/sdk/transforms/ViewTest.java   | 15 ---
 .../org/apache/beam/sdk/transforms/WatchTest.java  | 20 -
 .../sdk/transforms/display/DisplayDataTest.java|  6 +--
 .../transforms/windowing/CalendarWindowsTest.java  |  3 +-
 .../sdk/transforms/windowing/FixedWindowsTest.java |  3 +-
 .../beam/sdk/values/PCollectionListTest.java   |  3 +-
 .../beam/sdk/values/PCollectionTupleTest.java  |  3 +-
 .../operator/string/BeamSqlTrimExpressionTest.java |  9 ++--
 .../stream/HarnessStreamObserverFactories.java |  2 +-
 .../beam/sdk/io/aws/s3/S3ResourceIdTest.java   | 20 -
 .../java/org/apache/beam/sdk/io/amqp/AmqpIO.java   |  2 +-
 .../sdk/io/gcp/bigquery/BigQueryHelpersTest.java   | 12 +++---
 .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java   |  8 ++--
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 11 +++--
 .../beam/sdk/io/gcp/datastore/V1TestUtil.java  |  2 +-
 .../io/hadoop/inputformat/HIFIOCassandraIT.java|  2 +-
 .../hadoop/inputformat/HIFIOWithElasticTest.java   |  2 +-
 .../HIFIOWithEmbeddedCassandraTest.java|  2 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java|  3 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  | 17 
 .../apache/beam/sdk/io/mongodb/MongoDbIOTest.java  |  2 +-
 .../java/org/apache/beam/sdk/io/mqtt/MqttIO.java   |  2 +-
 .../java/org/apache/beam/sdk/io/tika/TikaIO.java   |  2 +-
 66 files changed, 206 insertions(+), 234 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Commented] (BEAM-4282) :beam-sdks-python-container:docker fails with cython error

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496851#comment-16496851
 ] 

Alan Myrvold commented on BEAM-4282:


Fixed in https://github.com/apache/beam/pull/5375

> :beam-sdks-python-container:docker fails with cython error
> --
>
> Key: BEAM-4282
> URL: https://issues.apache.org/jira/browse/BEAM-4282
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Started failing with [BEAM-3981] Futurize coders subpackage #5053
> ./gradlew :beam-sdks-python-container:docker --info
> Fails with:
> Might be due to the version of cython in the Dockerfile? "You are using 
> version 0.27.2 of cython. However, version 0.28.1 is recommended."
>  
>     /tmp/pip-req-build-77LZDu/setup.py:78: UserWarning: You are using version 
> 0.27.2 of cython. However, version 0.28.1 is recommended.
>       _CYTHON_VERSION, REQUIRED_CYTHON_VERSION
>     
>     Error compiling Cython file:
>     
>     ...
>     except NameError:   # Python 3
>       long = int
>       unicode = str
>     
>     
>     class CoderImpl(object):
>     ^
>     
>     
>     apache_beam/coders/coder_impl.py:69:0: 'object' is not a type name



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4432) Performance tests need a way to generate Synthetic data

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496870#comment-16496870
 ] 

Pablo Estrada commented on BEAM-4432:
-

Hi Ismael!
I understand your concerns. The code for these sources has specific knobs that 
will allow us to do specific testing of different scenarios. As an example, 
we're looking to study GroupByKey performance in different runners (specially 
for batch), and this requires testing a uniform distribution, but also testing 
a distribution that is heavily weighted towards a few keys (Zipf, one-key). 
We'll also work on tests for Combine, CoGroupByKey, ParDo (to understand 
per-fused-step overhead), etc.
Later on, as we decide to test dynamic rebalancing, we may want to have steps 
that produce artifical cpu-based or sleep-based delays (`SyntheticStep`), to 
see how runners react to different unexpected per-element processing delays.
I'd like to go forward with this, and focus next on adding a robust suite of 
perf tests

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4438) AvroCoder - Support Logical Types

[Nalseez Duke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nalseez Duke updated BEAM-4438:
---
Description: 
Context (from ASF Slack, BEAM channel):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}

  was:
Context (from ASF Slack, BEAM channel ):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}


> AvroCoder - Support Logical Types
> -
>
> Key: BEAM-4438
> URL: https://issues.apache.org/jira/browse/BEAM-4438
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Nalseez Duke
>Assignee: Kenneth Knowles
>Priority: Major
>
> Context (from ASF Slack, BEAM channel):
> {code:java}
> Hey all!
> So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
> Avro Logical Types 
> (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) are supported 
> within the AvroCoder.
> From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
> something. Does it make sense to allow for a user to supply a set of 
> Conversion(s) to be used within the ReflectData in the AvroCoder?
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4438) AvroCoder - Support Logical Types

[Nalseez Duke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nalseez Duke updated BEAM-4438:
---
Description: 
Context (from ASF Slack, BEAM channel ):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}

  was:
Context (from Apache BEAM Slack):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}


> AvroCoder - Support Logical Types
> -
>
> Key: BEAM-4438
> URL: https://issues.apache.org/jira/browse/BEAM-4438
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Nalseez Duke
>Assignee: Kenneth Knowles
>Priority: Major
>
> Context (from ASF Slack, BEAM channel ):
> {code:java}
> Hey all!
> So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
> Avro Logical Types 
> (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) are supported 
> within the AvroCoder.
> From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
> something. Does it make sense to allow for a user to supply a set of 
> Conversion(s) to be used within the ReflectData in the AvroCoder?
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4432) Performance tests need a way to generate Synthetic data

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496870#comment-16496870
 ] 

Pablo Estrada edited comment on BEAM-4432 at 5/31/18 5:16 PM:
--

Hi Ismael!
I understand your concerns. The code for these sources has specific knobs that 
will allow us to do specific testing of different scenarios. As an example, 
we're looking to study GroupByKey performance in different runners (specially 
for batch), and this requires testing a uniform distribution, but also testing 
a distribution that is heavily weighted towards a few keys (Zipf, one-key). 
We'll also work on tests for Combine, CoGroupByKey, ParDo (to understand 
per-fused-step overhead), etc.

Later on, as we decide to test dynamic rebalancing, we may want to have steps 
that produce artifical cpu-based or sleep-based delays (`SyntheticStep`), to 
see how runners react to different unexpected per-element processing delays.

I'd like to go forward with this, and focus next on adding a robust suite of 
perf tests. WDYT?


was (Author: pabloem):
Hi Ismael!
I understand your concerns. The code for these sources has specific knobs that 
will allow us to do specific testing of different scenarios. As an example, 
we're looking to study GroupByKey performance in different runners (specially 
for batch), and this requires testing a uniform distribution, but also testing 
a distribution that is heavily weighted towards a few keys (Zipf, one-key). 
We'll also work on tests for Combine, CoGroupByKey, ParDo (to understand 
per-fused-step overhead), etc.

Later on, as we decide to test dynamic rebalancing, we may want to have steps 
that produce artifical cpu-based or sleep-based delays (`SyntheticStep`), to 
see how runners react to different unexpected per-element processing delays.

I'd like to go forward with this, and focus next on adding a robust suite of 
perf tests

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107766
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:24
Start Date: 31/May/18 17:24
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393609217
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107766)
Time Spent: 18h  (was: 17h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107778
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:39
Start Date: 31/May/18 17:39
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393614048
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107778)
Time Spent: 18.5h  (was: 18h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #237

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1414dd56e79e73fe944e2af38db5730cd612e4e5 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1414dd56e79e73fe944e2af38db5730cd612e4e5
Commit message: "Merge pull request #4748: Avoid fully qualified class names, 
when there is an existing (static)…"
 > git rev-list --no-walk a0d1ad1c5e2aa89d458d609a83e9a0975e768ad9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5224045474307794833.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6846373940477497526.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4228457763046619599.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-237
Error from server (AlreadyExists): namespaces "filebasedioithdfs-237" already 
exists
Build step 'Execute shell' marked build as failure

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107856
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192217992
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/combine.go
 ##
 @@ -217,3 +224,162 @@ func (n *Combine) fail(err error) error {
 func (n *Combine) String() string {
return fmt.Sprintf("Combine[%v] Keyed:%v Out:%v", 
path.Base(n.Fn.Name()), n.UsesKey, n.Out.ID())
 }
+
+// The nodes below break apart the Combine into components to support
+// Combiner Lifting optimizations.
+
+// LiftedCombine is an executor for combining values before grouping by keys
+// for a lifted combine. Partially groups values by key within a bundle,
+// accumulating them in an in memory cache, before emitting them in the
+// FinishBundle step.
+type LiftedCombine struct {
+   *Combine
+
+   cache map[interface{}]FullValue
+}
+
+func (n *LiftedCombine) String() string {
+   return fmt.Sprintf("LiftedCombine[%v] Keyed:%v Out:%v", 
path.Base(n.Fn.Name()), n.UsesKey, n.Out.ID())
+}
+
+// StartBundle initializes the in memory cache of keys to accumulators.
+func (n *LiftedCombine) StartBundle(ctx context.Context, id string, data 
DataManager) error {
+   if err := n.Combine.StartBundle(ctx, id, data); err != nil {
+   return err
+   }
+   n.cache = make(map[interface{}]FullValue)
+   return nil
+}
+
+// ProcessElement takes a KV pair and combines values with the same into an 
accumulator,
+// caching them until the bundle is complete.
+func (n *LiftedCombine) ProcessElement(ctx context.Context, value FullValue, 
values ...ReStream) error {
+   if n.status != Active {
+   return fmt.Errorf("invalid status for precombine %v: %v", 
n.UID, n.status)
+   }
+
+   // Value is a KV so Elm & Elm2 are populated.
+   // Check the cache for an already present accumulator
+
+   afv, notfirst := n.cache[value.Elm]
+   var a interface{}
+   if notfirst {
+   a = afv.Elm2
+   } else {
+   b, err := n.newAccum(ctx, value.Elm)
+   if err != nil {
+   return n.fail(err)
+   }
+   a = b
+   }
+
+   a, err := n.addInput(ctx, a, value.Elm, value.Elm2, value.Timestamp, 
!notfirst)
+   if err != nil {
+   return n.fail(err)
+   }
+
+   // Cache the accumulator with the key
+   n.cache[value.Elm] = FullValue{Windows: value.Windows, Elm: value.Elm, 
Elm2: a, Timestamp: value.Timestamp}
 
 Review comment:
   I think it would be incredibly helpful if progress reporting could report 
the number of elements in a cache (as an SDK-specific counter (or standard 
runner counter?)). A data structure that grows without bounds, without some 
observability hook, concerns me.
   
   Not required for this review; filing a JIRA is fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107856)
Time Spent: 2h 50m  (was: 2h 40m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107859
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192210019
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/combine.go
 ##
 @@ -42,10 +42,12 @@ type Combine struct {
errerrorx.GuardedError
 }
 
+// ID returns the UnitID for this node.
 
 Review comment:
   Do we want to consider embedding a "base class" struct that provides these 
trivial behaviors? It reduces needless-looking code and would additionally 
improve the output of golint since these methods would have 1, not N, 
implementations.
   
   Not necessary to make this change for this PR, just wanted to consider it. 
Can open a JIRA if we agree it's worthwhile.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107859)
Time Spent: 3h 20m  (was: 3h 10m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107863
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192209548
 
 

 ##
 File path: sdks/go/pkg/beam/core/graph/edge.go
 ##
 @@ -317,9 +315,17 @@ func newDoFnNode(op Opcode, g *Graph, s *Scope, u *DoFn, 
in []*Node, typedefs ma
return edge, nil
 }
 
+// CombinePerKeyScope is the canonical name for the scope containing combine
+// composite. With Beam Portability, "primitive" composite transforms like
+// combine have their URNs & payloads attached to a high level scope, with a
+// default representation beneath. This const permits the translation layer
 
 Review comment:
   The use of this const...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107863)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107861
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192220524
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/combine_test.go
 ##
 @@ -60,9 +170,30 @@ func TestCombine(t *testing.T) {
if err := p.Down(context.Background()); err != nil {
t.Fatalf("down failed: %v", err)
}
+}
 
-   expected := makeKV(42, 21)
-   if !equalList(out.Elements, expected) {
-   t.Errorf("pardo(sumFn) = %v, want %v", 
extractKeyedValues(out.Elements...), extractKeyedValues(expected...))
+func mergeFn(a, b int) int {
+   return a + b
+}
+
+type MyCombine struct{}
+
+func (*MyCombine) AddInput(k int64, a int) int64 {
+   return k + int64(a)
+}
+
+func (*MyCombine) MergeAccumulators(a, b int64) int64 {
+   return a + b
+}
+
+func (*MyCombine) ExtractOutput(a int64) int {
+   return int(a)
+}
+
 
 Review comment:
   It'd be nice to have a more elaborate combiner with differentiated input, 
accumulator, and output types.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107861)
Time Spent: 3h 40m  (was: 3.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4438) AvroCoder - Support Logical Types

[Nalseez Duke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nalseez Duke updated BEAM-4438:
---
Summary: AvroCoder - Support Logical Types  (was: AvroCoder - Support 
LogicalTypes)

> AvroCoder - Support Logical Types
> -
>
> Key: BEAM-4438
> URL: https://issues.apache.org/jira/browse/BEAM-4438
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Nalseez Duke
>Assignee: Kenneth Knowles
>Priority: Major
>
> Context (from Apache BEAM Slack):
> {code:java}
> Hey all!
> So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
> Avro Logical Types 
> (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) are supported 
> within the AvroCoder.
> From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
> something. Does it make sense to allow for a user to supply a set of 
> Conversion(s) to be used within the ReflectData in the AvroCoder?
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #578

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4225) Integrate Nexmark with perfkit dashboards

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4225?focusedWorklogId=107774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107774
 ]

ASF GitHub Bot logged work on BEAM-4225:


Author: ASF GitHub Bot
Created on: 31/May/18 17:35
Start Date: 31/May/18 17:35
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #4976: [BEAM-4225] Add 
Nexmark PostCommit runs for spark, flink and direct runner and export to 
Bigquery
URL: https://github.com/apache/beam/pull/4976#issuecomment-393612627
 
 
   Created a new dataset named "nexmark" in apache-beam-testing


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107774)
Time Spent: 20m  (was: 10m)

> Integrate Nexmark with perfkit dashboards
> -
>
> Key: BEAM-4225
> URL: https://issues.apache.org/jira/browse/BEAM-4225
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The aim is to run Nexmark as post-commits and export the results to the 
> performance dashboards:
> see the threads:
> [https://lists.apache.org/thread.html/9f8fe1c6df7d8bfe2697332e69722ca4edd2810adc6a914cdf32da29@%3Cdev.beam.apache.org%3E]
> https://lists.apache.org/thread.html/701196efd6e74b7715785d43019a4a73e8a093997f59662fdadf8f2a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4438) AvroCoder - Support Logical Types

[Nalseez Duke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nalseez Duke updated BEAM-4438:
---
Description: 
Context (from ASF Slack, BEAM channel):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}
Potentially relevant Apache Avro Bug: 
https://issues.apache.org/jira/browse/AVRO-1891

  was:
Context (from ASF Slack, BEAM channel):
{code:java}
Hey all!

So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
Avro Logical Types (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) 
are supported within the AvroCoder.

>From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
>something. Does it make sense to allow for a user to supply a set of 
>Conversion(s) to be used within the ReflectData in the AvroCoder?
{code}


> AvroCoder - Support Logical Types
> -
>
> Key: BEAM-4438
> URL: https://issues.apache.org/jira/browse/BEAM-4438
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Nalseez Duke
>Assignee: Kenneth Knowles
>Priority: Major
>
> Context (from ASF Slack, BEAM channel):
> {code:java}
> Hey all!
> So I've been working with the BEAM 2.4.0 Java SDK, and I am trying to see if 
> Avro Logical Types 
> (https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types) are supported 
> within the AvroCoder.
> From what I'm seeing it doesn't seem to be the case, though maybe I'm missing 
> something. Does it make sense to allow for a user to supply a set of 
> Conversion(s) to be used within the ReflectData in the AvroCoder?
> {code}
> Potentially relevant Apache Avro Bug: 
> https://issues.apache.org/jira/browse/AVRO-1891



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107800=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107800
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 18:17
Start Date: 31/May/18 18:17
Worklog Time Spent: 10m 
  Work Description: swegner commented on a change in pull request #5521: 
[BEAM-4303] Enforce ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#discussion_r192191263
 
 

 ##
 File path: 
examples/java/src/main/java/org/apache/beam/examples/cookbook/TriggerExample.java
 ##
 @@ -393,12 +393,12 @@ public void processElement(ProcessContext c, 
BoundedWindow window) throws Except
   static class ExtractFlowInfo extends DoFn> {
 @ProcessElement
 public void processElement(ProcessContext c) throws Exception {
-  String[] laneInfo = c.element().split(",");
+  String[] laneInfo = c.element().split(",", -1);
   if ("timestamp".equals(laneInfo[0])) {
 // Header row
 return;
   }
-  if (laneInfo.length < 48) {
+  if (laneInfo.length < 50) {
 
 Review comment:
   Can you elaborate on what this change is for? I'm not sure what this magic 
number represents. If you are familiar, perhaps extract it out as a named 
constant that provides some context. i.e. `const int NUM_FIELDS = 50;`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107800)
Time Spent: 1h  (was: 50m)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2281) call SqlFunctions in operator implementation

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496960#comment-16496960
 ] 

Kenneth Knowles commented on BEAM-2281:
---

[~vectorijk]

> call SqlFunctions in operator implementation
> 
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Calcite has a collections of functions in 
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to 
> leverage when adding operators as {{BeamSqlExpression}}. 
> [~xumingming] [~app-tarush], any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #243

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
[...truncated 292.98 KB...]
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:181)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:160)
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:77)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.226.138.239:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #612

[Apache Jenkins Server]

See 


--
[...truncated 18.42 MB...]
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_10_02_18-8988452077267499825
May 31, 2018 5:02:19 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_10_02_18-8988452077267499825 with 0 
expected assertions.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:18.722Z: Autoscaling is enabled for job 
2018-05-31_10_02_18-8988452077267499825. The number of workers will be between 
1 and 1000.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:18.778Z: Autoscaling was automatically enabled for 
job 2018-05-31_10_02_18-8988452077267499825.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:21.723Z: Checking required Cloud APIs are enabled.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:21.971Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:26.695Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.125Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.421Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.462Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.754Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.807Z: Elided trivial flatten 
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.849Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.893Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.941Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:27.991Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:28.041Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 31, 2018 5:02:30 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:02:28.080Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=107773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107773
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 31/May/18 17:34
Start Date: 31/May/18 17:34
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-393595125
 
 
   Hey Luke @lukecwik ,commit 
https://github.com/apache/beam/pull/5467/commits/b88800a80242ee683c1fb1440b6071768e57f0a9
 fixed gcp-io build failures: https://scans.gradle.com/s/ak7vddh4gswwg/projects.
   
   If I change 
https://github.com/apache/beam/pull/5467/files#diff-76558bfd90b93d37ab9369862732ffd6R29
 into original one: shadowTest project(":beam-sdks-java-core"), build fails 
into the same reason: https://scans.gradle.com/s/srknbaqioyx3o/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107773)
Time Spent: 3h 40m  (was: 3.5h)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #334

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
[...truncated 118.69 KB...]
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)

May 31, 2018 6:06:09 PM org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.createTable(DatabaseTestHelper.java:46)
at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT.setUp(HadoopInputFormatIOIT.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107803
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 18:18
Start Date: 31/May/18 18:18
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5521: [BEAM-4303] Enforce 
ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#issuecomment-393626426
 
 
   Just one comment above, then everything else LGTM (looks good to me).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107803)
Time Spent: 1h 10m  (was: 1h)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107805
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 18:28
Start Date: 31/May/18 18:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393629804
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107805)
Time Spent: 18h 50m  (was: 18h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4432) Performance tests need a way to generate Synthetic data

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496870#comment-16496870
 ] 

Pablo Estrada edited comment on BEAM-4432 at 5/31/18 5:12 PM:
--

Hi Ismael!
I understand your concerns. The code for these sources has specific knobs that 
will allow us to do specific testing of different scenarios. As an example, 
we're looking to study GroupByKey performance in different runners (specially 
for batch), and this requires testing a uniform distribution, but also testing 
a distribution that is heavily weighted towards a few keys (Zipf, one-key). 
We'll also work on tests for Combine, CoGroupByKey, ParDo (to understand 
per-fused-step overhead), etc.

Later on, as we decide to test dynamic rebalancing, we may want to have steps 
that produce artifical cpu-based or sleep-based delays (`SyntheticStep`), to 
see how runners react to different unexpected per-element processing delays.

I'd like to go forward with this, and focus next on adding a robust suite of 
perf tests


was (Author: pabloem):
Hi Ismael!
I understand your concerns. The code for these sources has specific knobs that 
will allow us to do specific testing of different scenarios. As an example, 
we're looking to study GroupByKey performance in different runners (specially 
for batch), and this requires testing a uniform distribution, but also testing 
a distribution that is heavily weighted towards a few keys (Zipf, one-key). 
We'll also work on tests for Combine, CoGroupByKey, ParDo (to understand 
per-fused-step overhead), etc.
Later on, as we decide to test dynamic rebalancing, we may want to have steps 
that produce artifical cpu-based or sleep-based delays (`SyntheticStep`), to 
see how runners react to different unexpected per-element processing delays.
I'd like to go forward with this, and focus next on adding a robust suite of 
perf tests

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5468: Add a test to show Flatten Execution

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ab01229848487570ed7fb0b7418db9e3b92a733d
Merge: 1414dd5 4a347e1
Author: Thomas Groh 
AuthorDate: Thu May 31 11:07:34 2018 -0700

Merge pull request #5468: Add a test to show Flatten Execution

 .../portable/RemoteStageEvaluatorFactoryTest.java  | 65 ++
 1 file changed, 65 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #388

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

--
[...truncated 18.20 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 31, 2018 6:23:47 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 31, 2018 6:23:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531182344-49ebd836/output/results/staging/
May 31, 2018 6:23:47 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash dv-AOICw_j8KuoCA3EikDQ> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0531182344-49ebd836/output/results/staging/pipeline-dv-AOICw_j8KuoCA3EikDQ.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 6:23:49 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_11_23_48-8467665149338201439?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-31_11_23_48-8467665149338201439

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 31, 2018 6:23:49 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_11_23_48-8467665149338201439
May 31, 2018 6:23:49 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_11_23_48-8467665149338201439 with 1 
expected assertions.
May 31, 2018 6:24:01 

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107865
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192220206
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/combine_test.go
 ##
 @@ -60,9 +170,30 @@ func TestCombine(t *testing.T) {
if err := p.Down(context.Background()); err != nil {
t.Fatalf("down failed: %v", err)
}
+}
 
-   expected := makeKV(42, 21)
-   if !equalList(out.Elements, expected) {
-   t.Errorf("pardo(sumFn) = %v, want %v", 
extractKeyedValues(out.Elements...), extractKeyedValues(expected...))
+func mergeFn(a, b int) int {
+   return a + b
+}
+
+type MyCombine struct{}
 
 Review comment:
   Wouldn't hurt to add a comment indicating that int is the input type and 
int64 is the accumulator and output type. The test is an authoritative piece of 
code demonstrating the contract. That comment will help people write their own 
combiners more easily, I believe.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107865)
Time Spent: 4h  (was: 3h 50m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107860
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192221679
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
 ##
 @@ -34,9 +34,13 @@ import (
"github.com/golang/protobuf/ptypes"
 )
 
+// TODO(lostluck): 2018/05/28 Extract these from the canonical enums in 
beam_runner_api.proto
 
 Review comment:
   I have a code snippet that does most of this. Let me flesh it out and send 
it to you.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107860)
Time Spent: 3.5h  (was: 3h 20m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107862
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192217544
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/combine.go
 ##
 @@ -217,3 +224,162 @@ func (n *Combine) fail(err error) error {
 func (n *Combine) String() string {
return fmt.Sprintf("Combine[%v] Keyed:%v Out:%v", 
path.Base(n.Fn.Name()), n.UsesKey, n.Out.ID())
 }
+
+// The nodes below break apart the Combine into components to support
+// Combiner Lifting optimizations.
+
+// LiftedCombine is an executor for combining values before grouping by keys
+// for a lifted combine. Partially groups values by key within a bundle,
+// accumulating them in an in memory cache, before emitting them in the
+// FinishBundle step.
+type LiftedCombine struct {
+   *Combine
+
+   cache map[interface{}]FullValue
+}
+
+func (n *LiftedCombine) String() string {
+   return fmt.Sprintf("LiftedCombine[%v] Keyed:%v Out:%v", 
path.Base(n.Fn.Name()), n.UsesKey, n.Out.ID())
+}
+
+// StartBundle initializes the in memory cache of keys to accumulators.
+func (n *LiftedCombine) StartBundle(ctx context.Context, id string, data 
DataManager) error {
+   if err := n.Combine.StartBundle(ctx, id, data); err != nil {
+   return err
+   }
+   n.cache = make(map[interface{}]FullValue)
+   return nil
+}
+
+// ProcessElement takes a KV pair and combines values with the same into an 
accumulator,
+// caching them until the bundle is complete.
+func (n *LiftedCombine) ProcessElement(ctx context.Context, value FullValue, 
values ...ReStream) error {
+   if n.status != Active {
+   return fmt.Errorf("invalid status for precombine %v: %v", 
n.UID, n.status)
+   }
+
+   // Value is a KV so Elm & Elm2 are populated.
+   // Check the cache for an already present accumulator
+
+   afv, notfirst := n.cache[value.Elm]
+   var a interface{}
+   if notfirst {
+   a = afv.Elm2
+   } else {
+   b, err := n.newAccum(ctx, value.Elm)
+   if err != nil {
+   return n.fail(err)
+   }
+   a = b
+   }
+
+   a, err := n.addInput(ctx, a, value.Elm, value.Elm2, value.Timestamp, 
!notfirst)
+   if err != nil {
+   return n.fail(err)
+   }
+
+   // Cache the accumulator with the key
+   n.cache[value.Elm] = FullValue{Windows: value.Windows, Elm: value.Elm, 
Elm2: a, Timestamp: value.Timestamp}
+
+   return nil
+}
+
+// FinishBundle iterates through the cached key, accumulator pairs, and then
 
 Review comment:
   cached (key, accumulator) pairs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107862)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107858
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192221853
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go
 ##
 @@ -148,10 +148,58 @@ func (m *marshaller) addScopeTree(s *ScopeTree) string {
Inputs:diff(in, out),
Outputs:   diff(out, in),
}
+
+   m.updateIfCombineComposite(s, transform)
+
m.transforms[id] = transform
return id
 }
 
+// updateIfCombineComposite examines the scope tree and sets the PTransform 
Spec
+// to be a CombinePerKey with a CombinePayload if it's a liftable composite.
+// Beam Portability requires that composites
 
 Review comment:
   Unfinished comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107858)
Time Spent: 3h 10m  (was: 3h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107857
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192209339
 
 

 ##
 File path: sdks/go/pkg/beam/core/graph/edge.go
 ##
 @@ -317,9 +315,17 @@ func newDoFnNode(op Opcode, g *Graph, s *Scope, u *DoFn, 
in []*Node, typedefs ma
return edge, nil
 }
 
+// CombinePerKeyScope is the canonical name for the scope containing combine
 
 Review comment:
   nit: scope-containing. I was reading this wondering how a Go SDK scope 
applied here. That's not the case, I believe.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107857)
Time Spent: 3h  (was: 2h 50m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107864
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 20:10
Start Date: 31/May/18 20:10
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5507: 
[BEAM-4276] Add combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#discussion_r192221943
 
 

 ##
 File path: sdks/go/pkg/beam/transforms/top/top.go
 ##
 @@ -106,9 +106,9 @@ func validate(t typex.FullType, n int, less interface{}) {
funcx.MustSatisfy(less, funcx.Replace(sig, beam.TType, t.Type()))
 }
 
-// TODO(herohde) 5/25/2017: the accumulator should be serializable with a 
Coder.
-// We need a coder here, because the elements are generally code-able only. 
Until
-// then, we do not support combiner lifting.
+// TODO(herohde) 5/25/2017: BEAM- the accumulator should be serializable
 
 Review comment:
   Do we have a number to fill in here yet? It's been a year. :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107864)
Time Spent: 3h 50m  (was: 3h 40m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3773) [SQL] Investigate JDBC interface for Beam SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=107719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107719
 ]

ASF GitHub Bot logged work on BEAM-3773:


Author: ASF GitHub Bot
Created on: 31/May/18 15:59
Start Date: 31/May/18 15:59
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5506: [BEAM-3773] Build 
JDBC jar
URL: https://github.com/apache/beam/pull/5506#issuecomment-393581317
 
 
   This is green.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107719)
Time Spent: 10.5h  (was: 10h 20m)

> [SQL] Investigate JDBC interface for Beam SQL
> -
>
> Key: BEAM-3773
> URL: https://issues.apache.org/jira/browse/BEAM-3773
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> JDBC allows integration with a lot of third-party tools, e.g 
> [Zeppelin|https://zeppelin.apache.org/docs/0.7.0/manual/interpreters.html], 
> [sqlline|https://github.com/julianhyde/sqlline]. We should look into how 
> feasible it is to implement a JDBC interface for Beam SQL



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=107743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107743
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 31/May/18 16:29
Start Date: 31/May/18 16:29
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5507: [BEAM-4276] Add 
combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#issuecomment-393592119
 
 
   R: @herohde  @wcn3  Ping :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107743)
Time Spent: 2h 40m  (was: 2.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107768
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:28
Start Date: 31/May/18 17:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393610372
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107768)
Time Spent: 18h 10m  (was: 18h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #664

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
[...truncated 243.57 KB...]
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$2/GetPane/Map as step s94
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$2/RunChecks as step s95
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$2/VerifyAssertions/ParDo(DefaultConclude) as step s96
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Combine.globally(Top(Natural))/WithKeys/AddKeys/Map as step s97
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
Combine.globally(Top(Natural))/Combine.perKey(Top(Natural))/GroupByKey as step 
s98
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
Combine.globally(Top(Natural))/Combine.perKey(Top(Natural))/Combine.GroupedValues
 as step s99
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Combine.globally(Top(Natural))/Values/Values/Map as step s100
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
Combine.globally(Top(Natural))/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
 as step s101
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
Combine.globally(Top(Natural))/View.AsIterable/CreateDataflowView as step s102
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Combine.globally(Top(Natural))/CreateVoid/Read(CreateSource) 
as step s103
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Combine.globally(Top(Natural))/ProduceDefault as step s104
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Combine.globally(Top(Natural))/Flatten.PCollections as step 
s105
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/Window.Into()/Window.Assign as step 
s106
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$3/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s107
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s108
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$3/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s109
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s110
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s111
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/RewindowActuals/Window.Assign as step 
s112
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/KeyForDummy/AddKeys/Map as step s113
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$3/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s114
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/Create.Values/Read(CreateSource) as 
step s115
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$3/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s116
May 31, 2018 6:13:22 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=107798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107798
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 31/May/18 18:13
Start Date: 31/May/18 18:13
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-393624655
 
 
   Blocked until 2.5.0 release


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107798)
Time Spent: 22h 40m  (was: 22.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 22h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #617

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107753
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 16:50
Start Date: 31/May/18 16:50
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393598587
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107753)
Time Spent: 17h 20m  (was: 17h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107758
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:08
Start Date: 31/May/18 17:08
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393604112
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107758)
Time Spent: 17h 50m  (was: 17h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 17h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107776
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:37
Start Date: 31/May/18 17:37
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393613060
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107776)
Time Spent: 18h 20m  (was: 18h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #614

[Apache Jenkins Server]

See 


Changes:

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
[...truncated 17.90 MB...]

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 6:02:00 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_11_01_59-17045743811938485821?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-31_11_01_59-17045743811938485821

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 6:02:00 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_11_01_59-17045743811938485821
May 31, 2018 6:02:00 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_11_01_59-17045743811938485821 with 0 
expected assertions.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:01:59.358Z: Autoscaling is enabled for job 
2018-05-31_11_01_59-17045743811938485821. The number of workers will be between 
1 and 1000.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:01:59.392Z: Autoscaling was automatically enabled for 
job 2018-05-31_11_01_59-17045743811938485821.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:02.076Z: Checking required Cloud APIs are enabled.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:02.274Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:07.037Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:07.518Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:07.761Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:07.810Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.087Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.129Z: Elided trivial flatten 
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.179Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.226Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.264Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 6:02:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:02:08.312Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[beam] branch master updated (1414dd5 -> ab01229)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1414dd5  Merge pull request #4748: Avoid fully qualified class names, 
when there is an existing (static)…
 add 4a347e1  Add a test to show Flatten Execution
 new ab01229  Merge pull request #5468: Add a test to show Flatten Execution

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../portable/RemoteStageEvaluatorFactoryTest.java  | 65 ++
 1 file changed, 65 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (ab01229 -> 3ca1db5)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ab01229  Merge pull request #5468: Add a test to show Flatten Execution
 add 8a07b98  [SQL] Fix PubsubsJsonIT
 new 3ca1db5  Merge pull request #5525: [SQL] Fix PubsubsJsonIT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java| 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5525: [SQL] Fix PubsubsJsonIT

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3ca1db5922978200aef00be1fda579a0317d17ef
Merge: ab01229 8a07b98
Author: Kenn Knowles 
AuthorDate: Thu May 31 12:56:09 2018 -0700

Merge pull request #5525: [SQL] Fix PubsubsJsonIT

 .../sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java| 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=107733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107733
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 31/May/18 16:15
Start Date: 31/May/18 16:15
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #5489: 
[BEAM-4290] proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#discussion_r192155816
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -102,13 +99,19 @@ message ArtifactChunk {
   bytes data = 1;
 }
 
+message PutArtifactMetadata {
+  // (Required) An identifier for artifact staging session.
+  string artifact_staging_id = 1;
 
 Review comment:
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107733)
Time Spent: 4h  (was: 3h 50m)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #577

[Apache Jenkins Server]

See 


Changes:

[apilloud] Run test on shadowJar on a flag

[apilloud] [SQL] Build mega jar for JDBC

[apilloud] [SQL] JDBC Class Loader IT

--
[...truncated 22.67 MB...]
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-75
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-75 msg: [container-75] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-75 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-74
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-73
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-74 msg: [container-74] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-73 msg: [container-73] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-74 terminating.
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-73 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-63
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-63 msg: [container-63] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-63 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-69
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-68
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-69 msg: [container-69] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-71
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-69 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-68 msg: [container-68] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-68 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-71 msg: [container-71] Exiting heartbeat loop..
May 31, 2018 4:42:07 PM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-67
May 31, 2018 4:42:07 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-71 terminating.
May 31, 2018 4:42:07 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107754
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 16:53
Start Date: 31/May/18 16:53
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393599610
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107754)
Time Spent: 17.5h  (was: 17h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=107756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107756
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 31/May/18 17:02
Start Date: 31/May/18 17:02
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-393602497
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107756)
Time Spent: 17h 40m  (was: 17.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 17h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #237

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4303) Enforce ErrorProne analysis in examples project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4303?focusedWorklogId=107721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107721
 ]

ASF GitHub Bot logged work on BEAM-4303:


Author: ASF GitHub Bot
Created on: 31/May/18 16:02
Start Date: 31/May/18 16:02
Worklog Time Spent: 10m 
  Work Description: cademarkegard commented on issue #5521: [BEAM-4303] 
Enforce ErrorProne analysis in examples project
URL: https://github.com/apache/beam/pull/5521#issuecomment-393582377
 
 
   Great point. I can update the string splits to have a `limit` of `-1` 
instead and that will pass ErrorProne then.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 107721)
Time Spent: 40m  (was: 0.5h)

> Enforce ErrorProne analysis in examples project
> ---
>
> Key: BEAM-4303
> URL: https://issues.apache.org/jira/browse/BEAM-4303
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{examples-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-examples-java:assemble}}
> # Fix each ErrorProne warning from the {{examples/java}} project.
> # In {{examples/java/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #613

[Apache Jenkins Server]

See 


Changes:

[coheigea] Avoid fully qualified class names, when there is an existing (static)

--
[...truncated 17.79 MB...]

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 5:39:55 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_10_39_54-13317376516374713946?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-31_10_39_54-13317376516374713946

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 5:39:55 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_10_39_54-13317376516374713946
May 31, 2018 5:39:55 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_10_39_54-13317376516374713946 with 0 
expected assertions.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:39:54.462Z: Autoscaling is enabled for job 
2018-05-31_10_39_54-13317376516374713946. The number of workers will be between 
1 and 1000.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:39:54.502Z: Autoscaling was automatically enabled for 
job 2018-05-31_10_39_54-13317376516374713946.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:39:56.709Z: Checking required Cloud APIs are enabled.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:39:56.903Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:01.836Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.312Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.570Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.619Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.889Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.926Z: Elided trivial flatten 
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:02.973Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:03.018Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:03.063Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 5:40:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T17:40:03.110Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #615

[Apache Jenkins Server]

See 


--
[...truncated 17.91 MB...]
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 6:44:38 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-31_11_44_37-7960046258769325992?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-31_11_44_37-7960046258769325992

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 31, 2018 6:44:38 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-31_11_44_37-7960046258769325992
May 31, 2018 6:44:38 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-31_11_44_37-7960046258769325992 with 0 
expected assertions.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:37.535Z: Autoscaling is enabled for job 
2018-05-31_11_44_37-7960046258769325992. The number of workers will be between 
1 and 1000.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:37.578Z: Autoscaling was automatically enabled for 
job 2018-05-31_11_44_37-7960046258769325992.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:40.125Z: Checking required Cloud APIs are enabled.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:40.342Z: Checking permissions granted to controller 
Service Account.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:45.716Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:46.180Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:46.496Z: Expanding GroupByKey operations into 
optimizable parts.
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:46.553Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:46.900Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:46.961Z: Elided trivial flatten 
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:47.022Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:47.072Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:47.129Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 31, 2018 6:44:50 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-31T18:44:47.175Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema