[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Rui Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553828#comment-16553828
 ] 

Rui Wang commented on BEAM-4847:


[~janpeuker] provided extra information that might help in BEAM-4848 (which is 
a dup of this JIRA that I have already closed).

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553824#comment-16553824
 ] 

Rui Wang commented on BEAM-4848:


[~rdub]

Yep seems like it is duplicate. I should searched other components as well 
before creating this JIRA.

 

[~janpeuker]

I am closing this Jira for now and let's track the effort in BEAM-4847.

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-4848.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553732#comment-16553732
 ] 

Ryan Williams commented on BEAM-4847:
-

FWIW, java precommit has failed 3x 
([1|https://builds.apache.org/job/beam_PreCommit_Java_Commit/503/console], 
[2|https://builds.apache.org/job/beam_PreCommit_Java_Phrase/71/console], 
[3|https://builds.apache.org/job/beam_PreCommit_Java_Phrase/73/]) on 
[#5958|https://github.com/apache/beam/pull/5958] for what seems like this 
reason.

Could it be related to [the Jenkins upgrade that happened this 
weekend|https://lists.apache.org/thread.html/266ad449ae7e7225d24db18d2ee51e2bd5ef17f8f2edcebf726ca4ae@%3Cdev.beam.apache.org%3E]?

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Jan Peuker (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553705#comment-16553705
 ] 

Jan Peuker commented on BEAM-4848:
--

In case it helps: For me this even happens building locally regardless of 
building tests or not, on MacOS 10.13.6, JDK 1.8.0_151 consuming 3.24GB of RAM 
on a 16GB machine
Regular build fails with:
{{Execution failed for task ':rat'.}}
{{ > java.lang.OutOfMemoryError: Java heap space}}

Gradle check fails with:
{{Execution failed for task ':rat'.}}
{{> java.lang.OutOfMemoryError: GC overhead limit exceeded}}

Looking at the failures of rat, I wonder if it has something to do with parsing 
dependencies:
{{> Task :rat
{{skipping symbolic link   -- too many levels of symbolic links.}}
{{skipping symbolic link ...-- too many levels of symbolic links.}}
{{Expiring Daemon because JVM Tenured space is exhausted}}

More similar errors are in Python:
{{> Task :beam-sdks-python:lintPy27}}
{{Running flake8 for module apache_beam   gen_protos.py   setup.py
test_config.py:}}
{{Expiring Daemon because JVM Tenured space is exhausted}}

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553701#comment-16553701
 ] 

Ryan Williams commented on BEAM-4848:
-

probably a dupe of BEAM-4847 right?

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #768

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1094

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 17.36 MB...]
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.095Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.141Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.188Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Partial
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.225Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into 
SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.278Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.325Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.371Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.456Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.501Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.539Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.587Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForKeys/Read
Jul 24, 2018 1:11:11 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-24T01:11:01.634Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #461

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #802

[Apache Jenkins Server]

See 

[beam-site] branch asf-site updated (e26fcbe -> 6a757d6)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from e26fcbe  Prepare repository for deployment.
 add b8b6e43  Update quickstart-java.md
 add 4b85fc2  This closes #508
 new 6a757d6  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html | 5 -
 src/get-started/quickstart-java.md | 6 --
 2 files changed, 11 deletions(-)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6a757d6e7dd0d13d37fd6803807349375f71d284
Author: Mergebot 
AuthorDate: Tue Jul 24 00:24:05 2018 +

Prepare repository for deployment.
---
 content/get-started/quickstart-java/index.html | 5 -
 1 file changed, 5 deletions(-)

diff --git a/content/get-started/quickstart-java/index.html 
b/content/get-started/quickstart-java/index.html
index 2a3993a..15c6a77 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -319,11 +319,6 @@ d-7/19/2018  11:00 PMsubprocess
 
 
 
-PS mvn compile exec:java -D 
exec.mainClass=org.apache.beam.examples.WordCount `
- -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
-
-
-
 $ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
 

[beam-site] branch mergebot updated (55244af -> 4b85fc2)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 55244af  This closes #503
 add e26fcbe  Prepare repository for deployment.
 new b8b6e43  Update quickstart-java.md
 new 4b85fc2  This closes #508

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html | 93 +-
 src/get-started/quickstart-java.md |  6 --
 2 files changed, 91 insertions(+), 8 deletions(-)

[beam-site] 02/02: This closes #508

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4b85fc20a0360e52b81feb8e70bb46f2a74de84e
Merge: e26fcbe b8b6e43
Author: Mergebot 
AuthorDate: Tue Jul 24 00:20:48 2018 +

This closes #508

 src/get-started/quickstart-java.md | 6 --
 1 file changed, 6 deletions(-)

[beam-site] 01/02: Update quickstart-java.md

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit b8b6e4340a2787665484e44abcdad74df254dc97
Author: Rafael Fernandez 
AuthorDate: Mon Jul 23 15:29:29 2018 -0700

Update quickstart-java.md

Remove unnecessary tab
---
 src/get-started/quickstart-java.md | 6 --
 1 file changed, 6 deletions(-)

diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index 8c3cd93..12e98d9 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -133,12 +133,6 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
 ```
 
-{:.runner-direct-powershell}
-```
-PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
- -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
-```
-
 {:.runner-apex}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \

Build failed in Jenkins: beam_PreCommit_Java_Cron #139

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 7.48 MB...]
at 
org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc$MethodHandlers.invoke(ArtifactStagingServiceGrpc.java:320)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.halfClosed(ServerImpl.java:718)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.halfClose(InProcessTransport.java:633)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ForwardingClientStream.halfClose(ForwardingClientStream.java:67)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:408)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:272)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:254)
at 
org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls.asyncUnaryCall(ClientCalls.java:64)
at 
org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc$ArtifactStagingServiceStub.commitManifest(ArtifactStagingServiceGrpc.java:225)
at 
org.apache.beam.runners.direct.portable.artifact.LocalFileSystemArtifactStagerServiceTest.commitManifestWithMissingArtifactFails(LocalFileSystemArtifactStagerServiceTest.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
   

Jenkins build is back to normal : beam_PostCommit_Go_GradleBuild #514

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Spark #110

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 135.28 KB...]
Packing task ':beam-runners-core-java:compileJava'
:beam-runners-core-java:compileJava (Thread[Task worker for ':' Thread 
36,5,main]) completed. Took 7.889 secs.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 36,5,main]) 
started.

> Task :beam-runners-core-java:classes
Skipping task ':beam-runners-core-java:classes' as it has no actions.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 36,5,main]) 
completed. Took 0.0 secs.
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
36,5,main]) started.

> Task :beam-runners-core-java:shadowJar
Build cache key for task ':beam-runners-core-java:shadowJar' is 
38f650a731f27fca44583bde22fd55cf
Caching disabled for task ':beam-runners-core-java:shadowJar': Caching has not 
been enabled for the task
Task ':beam-runners-core-java:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 2.886s [2886ms]
Average Time/Jar: 1.443s [1443.0ms]
***
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
36,5,main]) completed. Took 3.307 secs.
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
36,5,main]) started.
:beam-runners-spark:compileJava (Thread[Task worker for ':' Thread 42,5,main]) 
started.

> Task :beam-sdks-java-harness:compileJava
Build cache key for task ':beam-sdks-java-harness:compileJava' is 
9669d87ca9bebc9064489d7a9728a320
Task ':beam-sdks-java-harness:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-sdks-java-harness:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-sdks-java-harness:compileJava'.
Compiling with error-prone compiler
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-sdks-java-harness:compileJava'
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
36,5,main]) completed. Took 6.401 secs.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 36,5,main]) 
started.

> Task :beam-sdks-java-harness:classes
Skipping task ':beam-sdks-java-harness:classes' as it has no actions.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 36,5,main]) 
completed. Took 0.0 secs.
:beam-sdks-java-harness:jar (Thread[Task worker for ':' Thread 36,5,main]) 
started.

> Task :beam-sdks-java-harness:jar
Build cache key for task ':beam-sdks-java-harness:jar' is 
f842674bc707a4cf40fe79398f88105f
Caching disabled for task ':beam-sdks-java-harness:jar': Caching has not been 
enabled for the task
Task ':beam-sdks-java-harness:jar' is not up-to-date because:
  No history is available.
:beam-sdks-java-harness:jar (Thread[Task worker for ':' Thread 36,5,main]) 
completed. Took 0.036 secs.
:beam-sdks-java-harness:shadowJar (Thread[Task worker for ':' Thread 
36,5,main]) started.

> Task :beam-sdks-java-harness:shadowJar
Build cache key for task ':beam-sdks-java-harness:shadowJar' is 
9a93881829319141b24f115f4e3732e3
Caching disabled for task ':beam-sdks-java-harness:shadowJar': Caching has not 
been enabled for the task
Task ':beam-sdks-java-harness:shadowJar' is not up-to-date because:
  No history is available.

> Task :beam-runners-spark:compileJava
Build cache key for task ':beam-runners-spark:compileJava' is 
89a904321d5340c6999e093bbaa50ebd
Task ':beam-runners-spark:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-runners-spark:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-runners-spark:compileJava'.
Compiling with error-prone compiler
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-runners-spark:compileJava'
:beam-runners-spark:compileJava (Thread[Task worker for ':' Thread 42,5,main]) 
completed. Took 18.838 secs.
:beam-runners-spark:classes (Thread[Task worker for ':' Thread 42,5,main]) 
started.

> Task :beam-runners-spark:classes
Skipping task ':beam-runners-spark:classes' as it has no actions.
:beam-runners-spark:classes (Thread[Task worker for ':' Thread 42,5,main]) 

Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow #18

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 136.15 KB...]
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-runners-core-java:compileJava'
:beam-runners-core-java:compileJava (Thread[Task worker for ':' Thread 
8,5,main]) completed. Took 9.927 secs.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 8,5,main]) 
started.

> Task :beam-runners-core-java:classes
Skipping task ':beam-runners-core-java:classes' as it has no actions.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 8,5,main]) 
completed. Took 0.0 secs.
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
44,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileJava
Build cache key for task ':beam-runners-google-cloud-dataflow-java:compileJava' 
is 8539d063b0b5bd0a417b0734b2bdeee3
Task ':beam-runners-google-cloud-dataflow-java:compileJava' is not up-to-date 
because:
  No history is available.
Custom actions are attached to task 
':beam-runners-google-cloud-dataflow-java:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-runners-google-cloud-dataflow-java:compileJava'.
Compiling with error-prone compiler
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-runners-google-cloud-dataflow-java:compileJava'
:beam-runners-google-cloud-dataflow-java:compileJava (Thread[Task worker for 
':' Thread 32,5,main]) completed. Took 10.287 secs.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 32,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:classes
Skipping task ':beam-runners-google-cloud-dataflow-java:classes' as it has no 
actions.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 32,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 32,5,main]) started.

> Task :beam-runners-core-java:shadowJar
Build cache key for task ':beam-runners-core-java:shadowJar' is 
fa0998987d7b62f334874407f965759f
Caching disabled for task ':beam-runners-core-java:shadowJar': Caching has not 
been enabled for the task
Task ':beam-runners-core-java:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 2.624s [2624ms]
Average Time/Jar: 1.312s [1312.0ms]
***
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
44,5,main]) completed. Took 3.148 secs.
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
44,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 8801a401d563460911434fe5e0e808ad
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Task ':beam-runners-google-cloud-dataflow-java:shadowJar' is not up-to-date 
because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 2.611s [2611ms]
Average Time/Jar: 1.3055s [1305.5ms]
***
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 32,5,main]) completed. Took 3.18 secs.

> Task :beam-sdks-java-harness:compileJava
Build cache key for task ':beam-sdks-java-harness:compileJava' is 
9669d87ca9bebc9064489d7a9728a320
Task ':beam-sdks-java-harness:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-sdks-java-harness:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-sdks-java-harness:compileJava'.
Compiling with error-prone compiler
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-sdks-java-harness:compileJava'
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
44,5,main]) completed. Took 6.056 secs.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 44,5,main]) 
started.

> Task 

Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Flink #111

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 136.18 KB...]
Packing task ':beam-runners-core-java:compileJava'
:beam-runners-core-java:compileJava (Thread[Task worker for ':' Thread 
23,5,main]) completed. Took 6.99 secs.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 23,5,main]) 
started.

> Task :beam-runners-core-java:classes
Skipping task ':beam-runners-core-java:classes' as it has no actions.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 23,5,main]) 
completed. Took 0.0 secs.
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
23,5,main]) started.

> Task :beam-runners-core-java:shadowJar
Build cache key for task ':beam-runners-core-java:shadowJar' is 
4beb476b4c8cdd40f765c133108ebd98
Caching disabled for task ':beam-runners-core-java:shadowJar': Caching has not 
been enabled for the task
Task ':beam-runners-core-java:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 2.019s [2019ms]
Average Time/Jar: 1.0095s [1009.5ms]
***
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
23,5,main]) completed. Took 2.275 secs.
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
23,5,main]) started.

> Task :beam-sdks-java-harness:compileJava
Build cache key for task ':beam-sdks-java-harness:compileJava' is 
9669d87ca9bebc9064489d7a9728a320
Task ':beam-sdks-java-harness:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-sdks-java-harness:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-sdks-java-harness:compileJava'.
Compiling with error-prone compiler
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-sdks-java-harness:compileJava'
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
23,5,main]) completed. Took 5.677 secs.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 23,5,main]) 
started.

> Task :beam-sdks-java-harness:classes
Skipping task ':beam-sdks-java-harness:classes' as it has no actions.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 23,5,main]) 
completed. Took 0.004 secs.
:beam-sdks-java-harness:jar (Thread[Task worker for ':' Thread 23,5,main]) 
started.

> Task :beam-sdks-java-harness:jar
Build cache key for task ':beam-sdks-java-harness:jar' is 
f842674bc707a4cf40fe79398f88105f
Caching disabled for task ':beam-sdks-java-harness:jar': Caching has not been 
enabled for the task
Task ':beam-sdks-java-harness:jar' is not up-to-date because:
  No history is available.
:beam-sdks-java-harness:jar (Thread[Task worker for ':' Thread 23,5,main]) 
completed. Took 0.036 secs.
:beam-sdks-java-harness:shadowJar (Thread[Task worker for ':' Thread 
23,5,main]) started.

> Task :beam-sdks-java-harness:shadowJar
Build cache key for task ':beam-sdks-java-harness:shadowJar' is 
7598034024c4e9f6e910966e34aa6d83
Caching disabled for task ':beam-sdks-java-harness:shadowJar': Caching has not 
been enabled for the task
Task ':beam-sdks-java-harness:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 42 (includes project)
Total Time: 35.745s [35745ms]
Average Time/Jar: 0.8510714285714001s [851.0714285714ms]
***
:beam-sdks-java-harness:shadowJar (Thread[Task worker for ':' Thread 
23,5,main]) completed. Took 36.177 secs.
:beam-runners-java-fn-execution:compileJava (Thread[Task worker for ':' Thread 
23,5,main]) started.

> Task :beam-runners-java-fn-execution:compileJava
Build cache key for task ':beam-runners-java-fn-execution:compileJava' is 
6576c771f8f2a1821c3ac110bdf2dc9b
Task ':beam-runners-java-fn-execution:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task 
':beam-runners-java-fn-execution:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-runners-java-fn-execution:compileJava'.
Compiling with error-prone compiler
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: 

Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Direct #114

[Apache Jenkins Server]

See 


Changes:

[amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew

[aaltay] Add bash script to automate "Preparation for GPG" (#6015)

--
[...truncated 129.38 KB...]
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-runners-core-construction-java:compileJava'
:beam-runners-core-construction-java:compileJava (Thread[Task worker for ':' 
Thread 21,5,main]) completed. Took 10.37 secs.
:beam-runners-core-construction-java:classes (Thread[Task worker for ':' Thread 
21,5,main]) started.

> Task :beam-runners-core-construction-java:classes
Skipping task ':beam-runners-core-construction-java:classes' as it has no 
actions.
:beam-runners-core-construction-java:classes (Thread[Task worker for ':' Thread 
21,5,main]) completed. Took 0.0 secs.
:beam-runners-core-construction-java:shadowJar (Thread[Task worker for ':' 
Thread 21,5,main]) started.

> Task :beam-runners-core-construction-java:shadowJar
Build cache key for task ':beam-runners-core-construction-java:shadowJar' is 
d05e2176f48c67962e3d9f52f913bd98
Caching disabled for task ':beam-runners-core-construction-java:shadowJar': 
Caching has not been enabled for the task
Task ':beam-runners-core-construction-java:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 1.916s [1916ms]
Average Time/Jar: 0.958s [958.0ms]
***
:beam-runners-core-construction-java:shadowJar (Thread[Task worker for ':' 
Thread 21,5,main]) completed. Took 2.162 secs.
:beam-runners-core-java:compileJava (Thread[Task worker for ':' Thread 
21,5,main]) started.

> Task :beam-runners-core-java:compileJava
Build cache key for task ':beam-runners-core-java:compileJava' is 
e8d5bcde4be2e1ddf7fd0a05a6974e83
Task ':beam-runners-core-java:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-runners-core-java:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-runners-core-java:compileJava'.
Compiling with error-prone compiler
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-runners-core-java:compileJava'
:beam-runners-core-java:compileJava (Thread[Task worker for ':' Thread 
21,5,main]) completed. Took 5.992 secs.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 21,5,main]) 
started.

> Task :beam-runners-core-java:classes
Skipping task ':beam-runners-core-java:classes' as it has no actions.
:beam-runners-core-java:classes (Thread[Task worker for ':' Thread 21,5,main]) 
completed. Took 0.0 secs.
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
21,5,main]) started.

> Task :beam-runners-core-java:shadowJar
Build cache key for task ':beam-runners-core-java:shadowJar' is 
7d310b603c8de23d512a27dbb642c0e8
Caching disabled for task ':beam-runners-core-java:shadowJar': Caching has not 
been enabled for the task
Task ':beam-runners-core-java:shadowJar' is not up-to-date because:
  No history is available.
***
GRADLE SHADOW STATS

Total Jars: 2 (includes project)
Total Time: 1.816s [1816ms]
Average Time/Jar: 0.908s [908.0ms]
***
:beam-runners-core-java:shadowJar (Thread[Task worker for ':' Thread 
21,5,main]) completed. Took 2.109 secs.
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
21,5,main]) started.

> Task :beam-sdks-java-harness:compileJava
Build cache key for task ':beam-sdks-java-harness:compileJava' is 
9669d87ca9bebc9064489d7a9728a320
Task ':beam-sdks-java-harness:compileJava' is not up-to-date because:
  No history is available.
Custom actions are attached to task ':beam-sdks-java-harness:compileJava'.
All input files are considered out-of-date for incremental task 
':beam-sdks-java-harness:compileJava'.
Compiling with error-prone compiler
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Packing task ':beam-sdks-java-harness:compileJava'
:beam-sdks-java-harness:compileJava (Thread[Task worker for ':' Thread 
21,5,main]) completed. Took 4.805 secs.
:beam-sdks-java-harness:classes (Thread[Task worker for ':' Thread 21,5,main]) 
started.

> Task :beam-sdks-java-harness:classes
Skipping task ':beam-sdks-java-harness:classes' as it has no actions.

[jira] [Commented] (BEAM-4850) Remove dependency on google-apitools package in Python SDK.

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553585#comment-16553585
 ] 

Ahmet Altay commented on BEAM-4850:
---

Charles assigned this to you, because you and some other were planning to 
update dependencies.

> Remove dependency on google-apitools package in Python SDK.
> ---
>
> Key: BEAM-4850
> URL: https://issues.apache.org/jira/browse/BEAM-4850
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Charles Chen
>Priority: Minor
>
> AFAIK google-apitools is not officially supported. For Google Cloud Platform 
> APIs such as Datastore, Cloud Storage or Pub/Sub, using [Cloud Client 
> Libraries for 
> Python|https://github.com/GoogleCloudPlatform/google-cloud-python] is 
> recommended. If that not possible, we may be able to use 
> [https://github.com/google/google-api-python-client] - officially supported, 
> but now in maintenance mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4850) Remove dependency on google-apitools package in Python SDK.

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-4850:
-

Assignee: Charles Chen  (was: Ahmet Altay)

> Remove dependency on google-apitools package in Python SDK.
> ---
>
> Key: BEAM-4850
> URL: https://issues.apache.org/jira/browse/BEAM-4850
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Charles Chen
>Priority: Minor
>
> AFAIK google-apitools is not officially supported. For Google Cloud Platform 
> APIs such as Datastore, Cloud Storage or Pub/Sub, using [Cloud Client 
> Libraries for 
> Python|https://github.com/GoogleCloudPlatform/google-cloud-python] is 
> recommended. If that not possible, we may be able to use 
> [https://github.com/google/google-api-python-client] - officially supported, 
> but now in maintenance mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553543#comment-16553543
 ] 

Thomas Weise commented on BEAM-4847:


I have noticed this issue ("Expiring Daemon because JVM Tenured space is 
exhausted") since the last few days and also don't see any recent successful 
pre-commit build at https://builds.apache.org/job/beam_PreCommit_Java_Phrase/

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=126346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126346
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 23/Jul/18 23:15
Start Date: 23/Jul/18 23:15
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5952: 
[BEAM-3744] Python PubSub API Fixes and Tests
URL: https://github.com/apache/beam/pull/5952#discussion_r204581110
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/pubsub.py
 ##
 @@ -155,45 +175,94 @@ def to_runner_api_parameter(self, context):
 return self.to_runner_api_pickled(context)
 
 
-class ReadStringsFromPubSub(PTransform):
-  """A ``PTransform`` for reading utf-8 string payloads from Cloud Pub/Sub.
-
-  Outputs elements of type ``unicode``, decoded from UTF-8.
+@deprecated(since='2.6.0', extra_message='Use ReadFromPubSub instead.')
+def ReadStringsFromPubSub(topic=None, subscription=None, id_label=None):
+  return _ReadStringsFromPubSub(topic, subscription, id_label)
 
-  This class is deprecated.
-  """
+class _ReadStringsFromPubSub(PTransform):
+  """This class is deprecated. Use ``ReadFromPubSub`` instead."""
 
   def __init__(self, topic=None, subscription=None, id_label=None):
-super(ReadStringsFromPubSub, self).__init__()
+super(_ReadStringsFromPubSub, self).__init__()
 self.topic = topic
 self.subscription = subscription
 self.id_label = id_label
 
   def expand(self, pvalue):
 p = (pvalue.pipeline
- | ReadFromPubSub(self.topic, self.subscription, self.id_label)
+ | ReadFromPubSub(self.topic, self.subscription, self.id_label,
+  with_attributes=False)
  | 'DecodeString' >> Map(lambda b: b.decode('utf-8')))
 p.element_type = basestring
 return p
 
 
-class WriteStringsToPubSub(PTransform):
-  """A ``PTransform`` for writing utf-8 string payloads to Cloud Pub/Sub."""
+@deprecated(since='2.6.0', extra_message='Use WriteToPubSub instead.')
+def WriteStringsToPubSub(topic):
+  return _WriteStringsToPubSub(topic)
+
+
+class _WriteStringsToPubSub(PTransform):
+  """This class is deprecated. Use ``WriteToPubSub`` instead."""
 
   def __init__(self, topic):
-"""Initializes ``WriteStringsToPubSub``.
+"""Initializes ``_WriteStringsToPubSub``.
 
 Attributes:
   topic: Cloud Pub/Sub topic in the form "/topics//".
 """
-super(WriteStringsToPubSub, self).__init__()
-self._sink = _PubSubPayloadSink(topic)
+super(_WriteStringsToPubSub, self).__init__()
+self._sink = _PubSubSink(topic, id_label=None, with_attributes=False,
+ timestamp_attribute=None)
 
   def expand(self, pcoll):
 pcoll = pcoll | 'EncodeString' >> Map(lambda s: s.encode('utf-8'))
 pcoll.element_type = bytes
 return pcoll | Write(self._sink)
 
+
+class WriteToPubSub(PTransform):
+  """A ``PTransform`` for writing messages to Cloud Pub/Sub."""
+  # Implementation note: This ``PTransform`` is overridden by Directrunner.
+
+  def __init__(self, topic, with_attributes, id_label=None,
+   timestamp_attribute=None):
+"""Initializes ``WriteToPubSub``.
+
+Args:
+  topic: Cloud Pub/Sub topic in the form "/topics//".
+  with_attributes:
+True - input elements will be :class:`~PubsubMessage` objects.
+False - input elements will be of type ``six.binary_type`` (message
+  data only).
+  id_label: If set, will set an attribute for each Cloud Pub/Sub message
+with the given name and a unique value. This attribute can then be used
+in a ReadFromPubSub PTransform to deduplicate messages.
+  timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
+message with the given name and the message's publish time as the 
value.
+"""
+super(WriteToPubSub, self).__init__()
+self.with_attributes = with_attributes
+self.id_label = id_label
+self.timestamp_attribute = timestamp_attribute
+self._sink = _PubSubSink(topic, id_label, with_attributes,
+ timestamp_attribute)
+
+  @staticmethod
+  def to_proto(element):
+if not isinstance(element, PubsubMessage):
+  raise TypeError('Unexpected element. Type: %s (expected: PubsubMessage), 
'
+  'value: %r' % (type(element), element))
+return element._to_proto()
+
+  def expand(self, pcoll):
+if self.with_attributes:
+  pcoll = pcoll | 'ToProtobuf' >> Map(self.to_proto)
+  pcoll.element_type = six.binary_type
 
 Review comment:
   After some investigation I believe that this is correct.
   The output of this mapping is a string (uses the proto object's 
`SerializeToString()`).
   On Dataflow, this string is passed to a runner harness written in Java, so 
it must be a serialized protobuf to be 

[jira] [Commented] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553523#comment-16553523
 ] 

Luke Cwik commented on BEAM-4819:
-

Yes I believe it should be.

 

Its unfortunate but a lot of stuff was baked into PipelineOptions at a top 
level which makes it annoying to use as a top level options class for 
configuring runners. A few ideas came to mind:
 * (preferred) Create a new top level interface called options which has the 
same capabilities as PipelineOptions (the "as" method and a few others) 
allowing PipelineOptions to be decoupled from Options which has all the 
argument parsing and JSON conversion magic.
 ** Good: PipelineOptions can still be passed around to things like FileSystem 
configuration, paves the way for transform level options, ...
 * Use string[] args and another argument parsing library building a simpler 
version of PipelineOptionsFactory/PipelineOptions dedicated to runners.
 ** Good: Likely fastest solution to implement

> Make portable Flink runner JobBundleFactory configurable
> 
>
> Key: BEAM-4819
> URL: https://issues.apache.org/jira/browse/BEAM-4819
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>
> BEAM-4791 introduces factory override for testing, expand that to allow users 
> to configure a different factory via service loader to adopt alternative 
> execution environments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553497#comment-16553497
 ] 

Thomas Weise commented on BEAM-4819:


CC: [~bsidhom], [~lcwik] this should be be based on the runner options.  

> Make portable Flink runner JobBundleFactory configurable
> 
>
> Key: BEAM-4819
> URL: https://issues.apache.org/jira/browse/BEAM-4819
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>
> BEAM-4791 introduces factory override for testing, expand that to allow users 
> to configure a different factory via service loader to adopt alternative 
> execution environments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4850) Remove dependency on google-apitools package in Python SDK.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-4850:
-

 Summary: Remove dependency on google-apitools package in Python 
SDK.
 Key: BEAM-4850
 URL: https://issues.apache.org/jira/browse/BEAM-4850
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay


AFAIK google-apitools is not officially supported. For Google Cloud Platform 
APIs such as Datastore, Cloud Storage or Pub/Sub, using [Cloud Client Libraries 
for Python|https://github.com/GoogleCloudPlatform/google-cloud-python] is 
recommended. If that not possible, we may be able to use 
[https://github.com/google/google-api-python-client] - officially supported, 
but now in maintenance mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553446#comment-16553446
 ] 

Rui Wang edited comment on BEAM-4848 at 7/23/18 10:26 PM:
--

Logs showed

`Expiring Daemon because JVM Tenured space is exhausted`

 

Seems like we need to tune GC args for each Jenkins build for Java and/or 
increase server ram size.


was (Author: amaliujia):
Logs showed

`Expiring Daemon because JVM Tenured space is exhausted`

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=126324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126324
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 23/Jul/18 22:16
Start Date: 23/Jul/18 22:16
Worklog Time Spent: 10m 
  Work Description: xinyuiscool opened a new pull request #6029: 
[BEAM-4849] Support running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029
 
 
   Add a couple of util classes and script to run beam job containers in Yarn:
   
   1) A factory class to read config and load into a static Samza job model;
   2) Supply the static model to the container runner.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126324)
Time Spent: 10m
Remaining Estimate: 0h

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>  

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126322
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 22:12
Start Date: 23/Jul/18 22:12
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407174324
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126322)
Time Spent: 4h 20m  (was: 4h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553446#comment-16553446
 ] 

Rui Wang commented on BEAM-4848:


Logs showed

```Expiring Daemon because JVM Tenured space is exhausted```

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553446#comment-16553446
 ] 

Rui Wang edited comment on BEAM-4848 at 7/23/18 10:02 PM:
--

Logs showed

`Expiring Daemon because JVM Tenured space is exhausted`


was (Author: amaliujia):
Logs showed

```Expiring Daemon because JVM Tenured space is exhausted```

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4778) Less wasteful ArtifactStagingService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4778?focusedWorklogId=126318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126318
 ]

ASF GitHub Bot logged work on BEAM-4778:


Author: ASF GitHub Bot
Created on: 23/Jul/18 21:43
Start Date: 23/Jul/18 21:43
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5958: [BEAM-4778] add 
option to flink job server to clean staged artifacts per-job
URL: https://github.com/apache/beam/pull/5958#issuecomment-407210812
 
 
   Looks like this is tracked at 
[BEAM-4847](https://issues.apache.org/jira/browse/BEAM-4847), per @lukecwik on 
Slack (thanks!)
   
   Not sure if I should keep trying here or wait until it seems like there's 
some movement on that issue?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126318)
Time Spent: 3.5h  (was: 3h 20m)

> Less wasteful ArtifactStagingService
> 
>
> Key: BEAM-4778
> URL: https://issues.apache.org/jira/browse/BEAM-4778
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java]
>  is the main implementation of ArtifactStagingService.
> It stages artifacts into a directory; and in practice the passed staging 
> session token is such that the directory is different for every job. This 
> leads to 2 issues:
>  * It doesn't get cleaned up when the job finishes or even when the 
> JobService shuts down, so we have disk space leaks if running a lot of jobs 
> (e.g. a suite of ValidatesRunner tests)
>  * We repeatedly re-stage the same artifacts. Instead, ideally, we should 
> identify that some artifacts don't need to be staged - based on knowing their 
> md5. The artifact staging protocol has rudimentary support for this but may 
> need to be modified.
> CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2661) Add KuduIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2661?focusedWorklogId=126311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126311
 ]

ASF GitHub Bot logged work on BEAM-2661:


Author: ASF GitHub Bot
Created on: 23/Jul/18 21:29
Start Date: 23/Jul/18 21:29
Worklog Time Spent: 10m 
  Work Description: timrobertson100 opened a new pull request #6021: 
[BEAM-2661] Adds KuduIO
URL: https://github.com/apache/beam/pull/6021
 
 
   Provides an implementation and tests for KuduIO.
   
   Please note that design decisions have been captured on 
[BEAM-2661](https://issues.apache.org/jira/browse/BEAM-2661).
   This implementation follows similar design patterns to `CassandraIO` and 
naming convention from `BigQueryIO`.
   
   The decision to use mocking and faking services for the unit tests was not 
taken lightly and will be replaced when Kudu offer an easier solution for Java 
- see [KUDU-2411](https://issues.apache.org/jira/browse/KUDU-2411)
   
   This implementation will benefit from the addition of authentication and the 
`BoundedSource` could be replaced by a `DoFn`. I propose adding those at a 
later date.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1093

[Apache Jenkins Server]

See 


--
[...truncated 15.55 MB...]
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.440Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.485Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.536Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write 
mutations to Cloud Spanner/Partition input
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.595Z: Fusing consumer SpannerIO.Write/To mutation 
group into ParDo(GenerateMutations)
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.627Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/GroupByWindow into 
SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.664Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Write into SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Reify
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:31.714Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/ParDo(ToIsmMetadataRecordForSize) into SpannerIO.Write/Write mutations to 
Cloud Spanner/Keys sample as view/GBKaSVForSize/Read
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.273Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.319Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.374Z: Starting 1 workers in us-central1-b...
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.374Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.414Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.451Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-23T21:16:32.503Z: Executing operation SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Create
Jul 23, 2018 9:16:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=126302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126302
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 23/Jul/18 20:51
Start Date: 23/Jul/18 20:51
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #6014: [BEAM-3709] 
Implementing new combine URNs in python.
URL: https://github.com/apache/beam/pull/6014#issuecomment-407196967
 
 
   I wasn't sure about tests here. There's no existing tests for any of these 
registered URNs and I don't have much experience in this codebase so adding 
tests for everything is a bit beyond me.
   
   For this change at least, the URNs I added are just the same implementations 
as the old Combine URNs, so it's at least code that's been used.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126302)
Time Spent: 40m  (was: 0.5h)

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, sdk-py-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Python SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126300
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 20:46
Start Date: 23/Jul/18 20:46
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407194973
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126300)
Time Spent: 4h 10m  (was: 4h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126299
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 20:46
Start Date: 23/Jul/18 20:46
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407195486
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126299)
Time Spent: 4h  (was: 3h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126298
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 20:44
Start Date: 23/Jul/18 20:44
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407194973
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126298)
Time Spent: 3h 50m  (was: 3h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2661) Add KuduIO

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson updated BEAM-2661:

Description: 
New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
design aspects documented below.
h2. The API

The {{KuduIO}} API requires the user to provide a function to convert objects 
into operations. This is similar to the {{JdbcIO}} but different to others, 
such as {{HBaseIO}} which requires a pre-transform stage beforehand to convert 
into the mutations to apply. It was originally intended to copy the {{HBaseIO}} 
approach, but this was not possible:
 # The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
 # An alternative could be to introduce a new object to beam (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
 ## It is not a familiar API to those already knowing Kudu
 ## It still requires serialization and deserialization of the operations. 
Using the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
 This is possible but has fragilities given the Kudu code itself continues to 
evolve. 
 ## It becomes a trivial codebase in Beam to maintain by defer the object to 
mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
precedent to do this.

h2. Testing framework

{{Kudu}} is written in C++. While a 
[TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
 does exist in Java, it requires binaries to be available for the target 
environment which is not portable (edit: this is now a [work in 
progress|https://issues.apache.org/jira/browse/KUDU-2411] in Kudu). Therefore 
we opt for the following:
 # Unit tests will use a mock Kudu client
 # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
Docker based Kudu instance

  was:
New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
design aspects documented below.
h2. The API

The {{KuduIO}} API requires the user to provide a function to convert objects 
into operations. This is similar to the {{JdbcIO}} but different to others, 
such as {{HBaseIO}} which requires a pre-transform stage beforehand to convert 
into the mutations to apply. It was originally intended to copy the {{HBaseIO}} 
approach, but this was not possible:
 # The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
 # An alternative could be to introduce a new object to beam (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
 ## It is not a familiar API to those already knowing Kudu
 ## It still requires serialization and deserialization of the operations. 
Using the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
 This is possible but has fragilities given the Kudu code itself continues to 
evolve. 
 ## It becomes a trivial codebase in Beam to maintain by defer the object to 
mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
precedent to do this.

h2. Testing framework

{{Kudu}} is written in C++. While a 
[TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
 does exist in Java, it requires binaries to be available for the target 
environment which is not portable. Therefore we opt for the following:
 # Unit tests will use a mock Kudu client
 # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
Docker based Kudu instance


> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: 

[jira] [Created] (BEAM-4849) Support running Beam Samza jobs in Yarn

[Xinyu Liu (JIRA)]

Xinyu Liu created BEAM-4849:
---

 Summary: Support running Beam Samza jobs in Yarn
 Key: BEAM-4849
 URL: https://issues.apache.org/jira/browse/BEAM-4849
 Project: Beam
  Issue Type: Improvement
  Components: runner-samza
Reporter: Xinyu Liu
Assignee: Xinyu Liu


We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-4847:

Summary: Java PreCommit is flaky: Expiring Daemon because JVM Tenured space 
is exhausted  (was: Java PreCommit is flaky)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553357#comment-16553357
 ] 

Luke Cwik commented on BEAM-4847:
-

This seems to be more of an existing issue: 
[https://lists.apache.org/thread.html/b47799450c73d1c2588a8029c224d1c85278fd7431568345a7928983@%3Cdev.beam.apache.org%3E]
 that [~swegner] dealt a lot with.

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4847:
---

Assignee: Alan Myrvold  (was: Luke Cwik)

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553355#comment-16553355
 ] 

Luke Cwik commented on BEAM-4847:
-

Expiring Daemon because JVM Tenured space is exhausted

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak reassigned BEAM-4686:
--

Assignee: (was: Melissa Pashniak)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Priority: Major
>  Labels: flake
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4848:
---
Description: Right now, Beam Java PreCommit and PostCommit tests are 
failing due GC issues.   (was: Right now, Beam Java PostCommit tests are 
failing due GC issues. )

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PreCommit and PostCommit tests are failing due GC 
> issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4848) GC overhead limit exceeded in Java Pre and PostCommit tests

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4848:
---
Summary: GC overhead limit exceeded in Java Pre and PostCommit tests  (was: 
GC overhead limit exceeded in Java PostCommit tests)

> GC overhead limit exceeded in Java Pre and PostCommit tests
> ---
>
> Key: BEAM-4848
> URL: https://issues.apache.org/jira/browse/BEAM-4848
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Rui Wang
>Priority: Major
>
> Right now, Beam Java PostCommit tests are failing due GC issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4848) GC overhead limit exceeded in Java PostCommit tests

[Rui Wang (JIRA)]

Rui Wang created BEAM-4848:
--

 Summary: GC overhead limit exceeded in Java PostCommit tests
 Key: BEAM-4848
 URL: https://issues.apache.org/jira/browse/BEAM-4848
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Rui Wang


Right now, Beam Java PostCommit tests are failing due GC issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126276
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 19:52
Start Date: 23/Jul/18 19:52
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #499: [BEAM-4802] Update 
the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407179981
 
 
   We see this error periodically -- try rerunning tests?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126276)
Time Spent: 2h 20m  (was: 2h 10m)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126277
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 19:52
Start Date: 23/Jul/18 19:52
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #499: [BEAM-4802] Update 
the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407180003
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126277)
Time Spent: 2.5h  (was: 2h 20m)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4847) Java PreCommit is flaky

[Boyuan Zhang (JIRA)]

Boyuan Zhang created BEAM-4847:
--

 Summary: Java PreCommit is flaky
 Key: BEAM-4847
 URL: https://issues.apache.org/jira/browse/BEAM-4847
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Boyuan Zhang
Assignee: Luke Cwik


Java PreCommit seems like flaky: 

https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126270
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 19:31
Start Date: 23/Jul/18 19:31
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407174324
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126270)
Time Spent: 3h 40m  (was: 3.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #138

[Apache Jenkins Server]

See 


--
[...truncated 10.61 MB...]
[2018-07-23T17:29:18,178][INFO ][o.e.n.Node   ] [node_sc6] 
closing ...
[2018-07-23T17:29:18,178][INFO ][o.e.c.s.ClusterService   ] [node_sd5] 
removed 
{{node_sc6}{C3QKd0nKTmKVrEHSepAm_Q}{BewkTuKnQGCu__q5Fmwicg}{local}{local[5]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [159]])
[2018-07-23T17:29:18,178][INFO ][o.e.c.s.ClusterService   ] [node_sm2] 
removed 
{{node_sc6}{C3QKd0nKTmKVrEHSepAm_Q}{BewkTuKnQGCu__q5Fmwicg}{local}{local[5]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [159]])
[2018-07-23T17:29:18,178][INFO ][o.e.c.s.ClusterService   ] [node_sd3] 
removed 
{{node_sc6}{C3QKd0nKTmKVrEHSepAm_Q}{BewkTuKnQGCu__q5Fmwicg}{local}{local[5]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [159]])
[2018-07-23T17:29:18,178][INFO ][o.e.c.s.ClusterService   ] [node_sm0] 
removed 
{{node_sc6}{C3QKd0nKTmKVrEHSepAm_Q}{BewkTuKnQGCu__q5Fmwicg}{local}{local[5]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [159]])
[2018-07-23T17:29:18,178][INFO ][o.e.c.s.ClusterService   ] [node_sd4] 
removed 
{{node_sc6}{C3QKd0nKTmKVrEHSepAm_Q}{BewkTuKnQGCu__q5Fmwicg}{local}{local[5]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [159]])
[2018-07-23T17:29:18,184][INFO ][o.e.n.Node   ] [node_sc6] 
closed
[2018-07-23T17:29:18,186][INFO ][o.e.n.Node   ] [node_sd3] 
stopping ...
[2018-07-23T17:29:18,190][INFO ][o.e.c.s.ClusterService   ] [node_sm1] 
removed 
{{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]},}, 
reason: 
zen-disco-node-left({node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]}),
 
reason(left)[{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]}
 left]
[2018-07-23T17:29:18,190][INFO ][o.e.n.Node   ] [node_sd3] 
stopped
[2018-07-23T17:29:18,190][INFO ][o.e.n.Node   ] [node_sd3] 
closing ...
[2018-07-23T17:29:18,191][INFO ][o.e.c.s.ClusterService   ] [node_sm2] 
removed 
{{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [160]])
[2018-07-23T17:29:18,191][INFO ][o.e.c.s.ClusterService   ] [node_sd4] 
removed 
{{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [160]])
[2018-07-23T17:29:18,191][INFO ][o.e.c.s.ClusterService   ] [node_sm0] 
removed 
{{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [160]])
[2018-07-23T17:29:18,191][INFO ][o.e.c.s.ClusterService   ] [node_sd5] 
removed 
{{node_sd3}{oC5eXR_OTmWyzcq4bulu5w}{DivKnhDFSuaV3N97KoGHWw}{local}{local[6]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [160]])
[2018-07-23T17:29:18,193][INFO ][o.e.n.Node   ] [node_sd3] 
closed
[2018-07-23T17:29:18,195][INFO ][o.e.n.Node   ] [node_sd4] 
stopping ...
[2018-07-23T17:29:18,199][INFO ][o.e.c.s.ClusterService   ] [node_sm1] 
removed 
{{node_sd4}{AMHbHU67S4qVQVkOn3UovQ}{eyyXHi6DR92ky1zOMgG0Pw}{local}{local[3]},}, 
reason: 
zen-disco-node-left({node_sd4}{AMHbHU67S4qVQVkOn3UovQ}{eyyXHi6DR92ky1zOMgG0Pw}{local}{local[3]}),
 
reason(left)[{node_sd4}{AMHbHU67S4qVQVkOn3UovQ}{eyyXHi6DR92ky1zOMgG0Pw}{local}{local[3]}
 left]
[2018-07-23T17:29:18,199][INFO ][o.e.n.Node   ] [node_sd4] 
stopped
[2018-07-23T17:29:18,199][INFO ][o.e.n.Node   ] [node_sd4] 
closing ...
[2018-07-23T17:29:18,200][INFO ][o.e.c.s.ClusterService   ] [node_sd5] 
removed 
{{node_sd4}{AMHbHU67S4qVQVkOn3UovQ}{eyyXHi6DR92ky1zOMgG0Pw}{local}{local[3]},}, 
reason: zen-disco-receive(from master [master 
{node_sm1}{23xmFrKhRDqhpotaBN4ksw}{ol73KG2pT5Wgs_TFBSlFdQ}{local}{local[1]} 
committed version [161]])
[2018-07-23T17:29:18,200][INFO ][o.e.c.s.ClusterService   ] [node_sm0] 
removed 
{{node_sd4}{AMHbHU67S4qVQVkOn3UovQ}{eyyXHi6DR92ky1zOMgG0Pw}{local}{local[3]},}, 

[beam] branch master updated: Add bash script to automate "Preparation for GPG" (#6015)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new ce4dbae  Add bash script to automate "Preparation for GPG" (#6015)
ce4dbae is described below

commit ce4dbaedb8d3f503b6df244ff9375c0013067717
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Mon Jul 23 12:03:43 2018 -0700

Add bash script to automate "Preparation for GPG" (#6015)

* Add bash script to automate "Preparation for GPG"
---
 .../src/main/scripts/preparation_before_release.sh | 101 +
 1 file changed, 101 insertions(+)

diff --git a/release/src/main/scripts/preparation_before_release.sh 
b/release/src/main/scripts/preparation_before_release.sh
new file mode 100755
index 000..4c3d5a9
--- /dev/null
+++ b/release/src/main/scripts/preparation_before_release.sh
@@ -0,0 +1,101 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will install and configure GPG key.
+
+set -e
+
+LOCAL_SVN_DIR=local_svn_dir
+ROOT_SVN_URL=https://dist.apache.org/repos/dist/
+DEV_REPO=dev
+RELEASE_REPO=release
+BEAM_REPO=beam
+
+cd ~
+
+echo "=Checking GPG Key"
+echo "You need a GPG key which reflects your Apache account."
+echo "Do you want to generate a new GPG key associated with your Apache 
account? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+  echo "===Generating new GPG key"
+  sudo apt-get install rng-tools
+  sudo rngd -r /dev/urandom
+  gpg --full-generate-key
+fi
+
+echo "Listing all GPG keys="
+gpg --list-keys
+echo "Please copy the public key which is associated with your Apache account:"
+read pub_key
+
+echo "===Configuring git signing key==="
+git config --global user.signingkey $pub_key
+git config --list
+
+echo "===Adding your key into KEYS file"
+echo "It's required to append your key into KEYS file in dist.apache.org"
+echo "Have you put your key in KEYS? [y|N]"
+read confirmation
+if [[ $confirmation != "y" ]]; then
+  echo "Only PMC member can write into dist.apache.org. Are you a PMC member? 
[y|N]"
+  read pmc_permission
+  if [[ $pmc_permission != "y" ]]; then
+echo "Please ask a PMC member to help you add your key in dev@ list."
+echo "Skip adding key into dist.apache.org/KEYS file."
+  else
+echo "Please input your name: "
+read name
+echo "==Starting updating KEYS file in dev repo"
+mkdir ${LOCAL_SVN_DIR}
+cd ${LOCAL_SVN_DIR}
+svn co ${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}
+cd ${BEAM_REPO}
+(gpg --list-sigs ${name} && gpg --armor --export ${name}) >> KEYS
+svn status
+echo "Please review all changes. Do you confirm to commit? [y|N]"
+read commit_confirmation
+if [[ $commit_confirmation = "y" ]]; then
+  svn commit KEYS
+else
+  echo "Not commit new changes into 
${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}${DEV_REPO}/KEYS"
+fi
+
+cd ~/${LOCAL_SVN_DIR}
+echo "===Starting updating KEYS file in release repo==="
+svn co ${ROOT_SVN_URL}/${RELEASE_REPO}/${BEAM_REPO}
+cd ${BEAM_REPO}
+(gpg --list-sigs ${name} && gpg --armor --export ${name}) >> KEYS
+svn status
+echo "Please review all changes. Do you confirm to commit? [y|N]"
+read commit_confirmation
+if [[ $commit_confirmation = "y" ]]; then
+  svn commit KEYS
+else
+  echo "Not commit new changes into 
${ROOT_SVN_URL}/${DEV_REPO}/${BEAM_REPO}${RELEASE_REPO}/KEYS"
+fi
+
+cd ~
+rm -rf ${LOCAL_SVN_DIR}
+  fi
+fi
+
+echo "Setting up gpg agent="
+eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)
+export GPG_TTY=$(tty)
+export GPG_AGENT_INFO

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e26fcbe915094063eae2d65f1dd9dfa40bdaac54
Author: Mergebot 
AuthorDate: Mon Jul 23 18:57:37 2018 +

Prepare repository for deployment.
---
 content/get-started/quickstart-java/index.html | 93 +-
 1 file changed, 91 insertions(+), 2 deletions(-)

diff --git a/content/get-started/quickstart-java/index.html 
b/content/get-started/quickstart-java/index.html
index eb293de..2a3993a 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -229,7 +229,7 @@ limitations under the License.
 
 The easiest way to get a copy of the WordCount pipeline is to use the 
following command to generate a simple Maven project that contains Beam’s 
WordCount examples and builds against the most recent Beam release:
 
-$ mvn 
archetype:generate \
+$ mvn 
archetype:generate \
   -DarchetypeGroupId=org.apache.beam \
   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
   -DarchetypeVersion=2.5.0 \
@@ -241,9 +241,21 @@ limitations under the License.
 
 
 
+PS mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion=2.5.0 `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+
+
+
 This will create a directory word-count-beam that contains a simple pom.xml and a series of example pipelines that 
count words in text files.
 
-$ cd 
word-count-beam/
+$ cd 
word-count-beam/
 
 $ ls
 pom.xmlsrc
@@ -254,6 +266,32 @@ MinimalWordCount.java  WordCount.java
 
 
 
+PS cd .\word-count-beam
+
+PS dir
+
+... 
+ 
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMsrc   
  
+-a7/19/2018  11:00 PM  16051 pom.xml
+
+PS dir .\src\main\java\org\apache\beam\examples
+
+...
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMcommon
  
+d-7/19/2018  11:00 PMcomplete  
  
+d-7/19/2018  11:00 PMsubprocess
  
+-a7/19/2018  11:00 PM   7073 DebuggingWordCount.java   
  
+-a7/19/2018  11:00 PM   5945 MinimalWordCount.java 
  
+-a7/19/2018  11:00 PM   9490 WindowedWordCount.java
  
+-a7/19/2018  11:00 PM   7662 WordCount.java
+
+
+
 For a detailed introduction to the Beam concepts used in these examples, 
see the WordCount Example 
Walkthrough. Here, we’ll just focus on executing WordCount.java.
 
 Run WordCount
@@ -274,11 +312,18 @@ MinimalWordCount.java WordCount.java
   Run your first WordCount pipeline.
 
 
+For Unix shells:
+
 $ 
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
 
 
 
+PS mvn compile exec:java -D 
exec.mainClass=org.apache.beam.examples.WordCount `
+ -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
+
+
+
 $ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
 
@@ -316,6 +361,50 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=/tmp/counts 
--runner=SamzaRunner" -Psamza-runner
 
 
+For Windows PowerShell:
+
+PS mvn compile exec:java -D 
exec.mainClass=org.apache.beam.examples.WordCount `
+ -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
+
+
+
+PS 
mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount `
+ -D exec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" -P 
apex-runner
+
+
+
+PS mvn compile exec:java -D 
exec.mainClass=org.apache.beam.examples.WordCount `
+ -D exec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" -P 
flink-runner
+
+
+
+PS mvn package exec:java -D 
exec.mainClass=org.apache.beam.examples.WordCount `
+ -D exec.args="--runner=FlinkRunner 

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=126239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126239
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:57
Start Date: 23/Jul/18 18:57
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #503: [BEAM-4471] Update 
Java Quickstart for Windows
URL: https://github.com/apache/beam-site/pull/503
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index acdcd2976..8c3cd9383 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -41,6 +41,7 @@ This Quickstart will walk you through executing your first 
Beam pipeline to run
 
 The easiest way to get a copy of the WordCount pipeline is to use the 
following command to generate a simple Maven project that contains Beam's 
WordCount examples and builds against the most recent Beam release:
 
+{:.shell-unix}
 ```
 $ mvn archetype:generate \
   -DarchetypeGroupId=org.apache.beam \
@@ -53,8 +54,23 @@ $ mvn archetype:generate \
   -DinteractiveMode=false
 ```
 
+{:.shell-PowerShell}
+```
+PS> mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{ site.release_latest }} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+```
+
+
 This will create a directory `word-count-beam` that contains a simple 
`pom.xml` and a series of example pipelines that count words in text files.
 
+{:.shell-unix}
 ```
 $ cd word-count-beam/
 
@@ -66,6 +82,33 @@ DebuggingWordCount.java  WindowedWordCount.java  common
 MinimalWordCount.java  WordCount.java
 ```
 
+{:.shell-PowerShell}
+```
+PS> cd .\word-count-beam
+
+PS> dir
+
+... 
+ 
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMsrc   
  
+-a7/19/2018  11:00 PM  16051 pom.xml
+
+PS> dir .\src\main\java\org\apache\beam\examples
+
+...
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMcommon
  
+d-7/19/2018  11:00 PMcomplete  
  
+d-7/19/2018  11:00 PMsubprocess
  
+-a7/19/2018  11:00 PM   7073 DebuggingWordCount.java   
  
+-a7/19/2018  11:00 PM   5945 MinimalWordCount.java 
  
+-a7/19/2018  11:00 PM   9490 WindowedWordCount.java
  
+-a7/19/2018  11:00 PM   7662 WordCount.java
+```
+
 For a detailed introduction to the Beam concepts used in these examples, see 
the [WordCount Example Walkthrough]({{ site.baseurl 
}}/get-started/wordcount-example). Here, we'll just focus on executing 
`WordCount.java`.
 
 
@@ -82,12 +125,20 @@ After you've chosen which runner you'd like to use:
 1. Choosing input files and an output location are accessible on the 
chosen runner. (For example, you can't access a local file if you are running 
the pipeline on an external cluster.)
 1.  Run your first WordCount pipeline.
 
+For Unix shells:
+
 {:.runner-direct}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
 ```
 
+{:.runner-direct-powershell}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
+```
+
 {:.runner-apex}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
@@ -131,6 +182,57 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=/tmp/counts 

[beam-site] branch asf-site updated (7cbd122 -> e26fcbe)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 7cbd122  Prepare repository for deployment.
 add 0a13ed8  Update Java Quickstart for Windows
 add 580b7eb  Update quickstart-java.md
 add 55244af  This closes #503
 new e26fcbe  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html |  93 +-
 src/get-started/quickstart-java.md | 102 +
 2 files changed, 193 insertions(+), 2 deletions(-)

[beam-site] 01/03: Update Java Quickstart for Windows

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 0a13ed8fa0795b8c66ac955265a81966c72547e0
Author: Rafael Fernandez 
AuthorDate: Thu Jul 19 16:40:37 2018 -0700

Update Java Quickstart for Windows

Add PowerShell options for setup, build, and direct runner 
execution/inspection to better support Microsoft Windows users.

Really the main difference is the need to have spaces after the "-D" and 
"-P"s in mvn, and break lines in PowerShell style.
---
 src/get-started/quickstart-java.md | 65 ++
 1 file changed, 65 insertions(+)

diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index acdcd29..4d22401 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -41,6 +41,7 @@ This Quickstart will walk you through executing your first 
Beam pipeline to run
 
 The easiest way to get a copy of the WordCount pipeline is to use the 
following command to generate a simple Maven project that contains Beam's 
WordCount examples and builds against the most recent Beam release:
 
+{:.unix}
 ```
 $ mvn archetype:generate \
   -DarchetypeGroupId=org.apache.beam \
@@ -52,9 +53,23 @@ $ mvn archetype:generate \
   -Dpackage=org.apache.beam.examples \
   -DinteractiveMode=false
 ```
+{:.powershell}
+```
+PS> mvn archetype:generate `
+ -D archetypeGroupId=org.apache.beam `
+ -D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
+ -D archetypeVersion={{ site.release_latest }} `
+ -D groupId=org.example `
+ -D artifactId=word-count-beam `
+ -D version="0.1" `
+ -D package=org.apache.beam.examples `
+ -D interactiveMode=false
+```
+
 
 This will create a directory `word-count-beam` that contains a simple 
`pom.xml` and a series of example pipelines that count words in text files.
 
+{:.unix}
 ```
 $ cd word-count-beam/
 
@@ -65,6 +80,32 @@ $ ls src/main/java/org/apache/beam/examples/
 DebuggingWordCount.javaWindowedWordCount.java  common
 MinimalWordCount.java  WordCount.java
 ```
+{:.powershell}
+```
+PS> cd .\word-count-beam
+
+PS> dir
+
+... 
+ 
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMsrc   
  
+-a7/19/2018  11:00 PM  16051 pom.xml
+
+PS> dir .\src\main\java\org\apache\beam\examples
+
+...
+ModeLastWriteTime Length Name  
  
+- --   
  
+d-7/19/2018  11:00 PMcommon
  
+d-7/19/2018  11:00 PMcomplete  
  
+d-7/19/2018  11:00 PMsubprocess
  
+-a7/19/2018  11:00 PM   7073 DebuggingWordCount.java   
  
+-a7/19/2018  11:00 PM   5945 MinimalWordCount.java 
  
+-a7/19/2018  11:00 PM   9490 WindowedWordCount.java
  
+-a7/19/2018  11:00 PM   7662 WordCount.java
+```
 
 For a detailed introduction to the Beam concepts used in these examples, see 
the [WordCount Example Walkthrough]({{ site.baseurl 
}}/get-started/wordcount-example). Here, we'll just focus on executing 
`WordCount.java`.
 
@@ -88,6 +129,12 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
 ```
 
+{:.runner-direct-powershell}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
+```
+
 {:.runner-apex}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
@@ -141,6 +188,11 @@ Once the pipeline has completed, you can view the output. 
You'll notice that the
 $ ls counts*
 ```
 
+{:.runner-direct-powershell}
+```
+PS> dir counts*
+```
+
 {:.runner-apex}
 ```
 $ ls counts*
@@ -186,6 +238,19 @@ Foundation: 1
 ...
 ```
 
+{:.runner-direct-powershell}
+```
+PS> cat counts*
+the: 28
+executions: 2
+available: 5
+project: 6
+clients: 4
+to: 11
+Dependencies: 1
+...
+```
+
 {:.runner-apex}
 ```
 $ cat counts*

[beam-site] branch mergebot updated (3bad728 -> 55244af)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 3bad728  This closes #506
 add 7cbd122  Prepare repository for deployment.
 new 0a13ed8  Update Java Quickstart for Windows
 new 580b7eb  Update quickstart-java.md
 new 55244af  This closes #503

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/portability/index.html |  32 +-
 src/get-started/quickstart-java.md| 102 ++
 2 files changed, 131 insertions(+), 3 deletions(-)

[beam-site] 02/03: Update quickstart-java.md

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 580b7ebb8152b4e56ef566b23beae85792c00c4d
Author: Rafael Fernandez 
AuthorDate: Fri Jul 20 15:14:08 2018 -0700

Update quickstart-java.md
---
 src/get-started/quickstart-java.md | 81 +++---
 1 file changed, 59 insertions(+), 22 deletions(-)

diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index 4d22401..8c3cd93 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -41,7 +41,7 @@ This Quickstart will walk you through executing your first 
Beam pipeline to run
 
 The easiest way to get a copy of the WordCount pipeline is to use the 
following command to generate a simple Maven project that contains Beam's 
WordCount examples and builds against the most recent Beam release:
 
-{:.unix}
+{:.shell-unix}
 ```
 $ mvn archetype:generate \
   -DarchetypeGroupId=org.apache.beam \
@@ -53,7 +53,8 @@ $ mvn archetype:generate \
   -Dpackage=org.apache.beam.examples \
   -DinteractiveMode=false
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> mvn archetype:generate `
  -D archetypeGroupId=org.apache.beam `
@@ -69,7 +70,7 @@ PS> mvn archetype:generate `
 
 This will create a directory `word-count-beam` that contains a simple 
`pom.xml` and a series of example pipelines that count words in text files.
 
-{:.unix}
+{:.shell-unix}
 ```
 $ cd word-count-beam/
 
@@ -80,7 +81,8 @@ $ ls src/main/java/org/apache/beam/examples/
 DebuggingWordCount.javaWindowedWordCount.java  common
 MinimalWordCount.java  WordCount.java
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> cd .\word-count-beam
 
@@ -123,6 +125,8 @@ After you've chosen which runner you'd like to use:
 1. Choosing input files and an output location are accessible on the 
chosen runner. (For example, you can't access a local file if you are running 
the pipeline on an external cluster.)
 1.  Run your first WordCount pipeline.
 
+For Unix shells:
+
 {:.runner-direct}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
@@ -178,6 +182,57 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=pom.xml --output=/tmp/counts 
--runner=SamzaRunner" -Psamza-runner
 ```
+For Windows PowerShell:
+
+{:.runner-direct}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--inputFile=pom.xml --output=counts" -P direct-runner
+```
+
+{:.runner-apex}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" -P 
apex-runner
+```
+
+{:.runner-flink-local}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" -P 
flink-runner
+```
+
+{:.runner-flink-cluster}
+```
+PS> mvn package exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=.\target\word-count-beam-bundled-0.1.jar `
+   --inputFile=C:\path\to\quickstart\pom.xml 
--output=C:\tmp\counts" -P flink-runner
+
+You can monitor the running job by visiting the Flink dashboard at 
http://:8081
+```
+
+{:.runner-spark}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" -P 
spark-runner
+```
+
+{:.runner-dataflow}
+```
+Make sure you complete the setup steps at 
https://beam.apache.org/documentation/runners/dataflow/#setup
+
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--runner=DataflowRunner --project= `
+   --gcpTempLocation=gs:///tmp `
+   --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" `
+ -P dataflow-runner
+```
+
+{:.runner-samza-local}
+```
+PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount 
`
+ -D exec.args="--inputFile=pom.xml --output=/tmp/counts 
--runner=SamzaRunner" -P samza-runner
+```
 
 ## Inspect the results
 
@@ -188,11 +243,6 @@ Once the pipeline has completed, you can view the output. 
You'll notice that the
 $ ls counts*
 ```
 
-{:.runner-direct-powershell}
-```
-PS> dir counts*
-```
-
 {:.runner-apex}
 ```
 $ ls counts*
@@ -238,19 +288,6 @@ Foundation: 1
 ...
 ```
 
-{:.runner-direct-powershell}
-```
-PS> cat counts*
-the: 28
-executions: 2
-available: 5
-project: 6
-clients: 4
-to: 11
-Dependencies: 1
-...
-```
-
 {:.runner-apex}
 ```
 $ cat counts*

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=126237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126237
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:53
Start Date: 23/Jul/18 18:53
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #503: [BEAM-4471] Update Java 
Quickstart for Windows
URL: https://github.com/apache/beam-site/pull/503#issuecomment-407163646
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126237)
Time Spent: 50m  (was: 40m)

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4831) Fix broken go vet task in the gradle build

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4831?focusedWorklogId=126232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126232
 ]

ASF GitHub Bot logged work on BEAM-4831:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:50
Start Date: 23/Jul/18 18:50
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6012: [BEAM-4831] Ignore 
failures during :beam-sdks-go:vet
URL: https://github.com/apache/beam/pull/6012#issuecomment-407162567
 
 
   I merged for now, the JIRA is open and can move this forward.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126232)
Time Spent: 0.5h  (was: 20m)

> Fix broken go vet task in the gradle build
> --
>
> Key: BEAM-4831
> URL: https://issues.apache.org/jira/browse/BEAM-4831
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Reproduce by running
> {{./gradlew check}}
> {{Today this is failing.}}
> {{go vet seems to be trying to validate files in a vendor folder.}}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4831) Fix broken go vet task in the gradle build

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4831?focusedWorklogId=126231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126231
 ]

ASF GitHub Bot logged work on BEAM-4831:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:49
Start Date: 23/Jul/18 18:49
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6012: [BEAM-4831] Ignore 
failures during :beam-sdks-go:vet
URL: https://github.com/apache/beam/pull/6012
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/build.gradle b/sdks/go/build.gradle
index 43386995d86..fe117bb9b69 100644
--- a/sdks/go/build.gradle
+++ b/sdks/go/build.gradle
@@ -35,4 +35,9 @@ golang {
 // Build integration test driver
 go 'build -o ./build/bin/integration 
github.com/apache/beam/sdks/go/test/integration'
   }
+
+  // Ignore spurious vet errors during check for [BEAM-4831].
+  vet {
+continueOnFailure = true
+  }
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126231)
Time Spent: 20m  (was: 10m)

> Fix broken go vet task in the gradle build
> --
>
> Key: BEAM-4831
> URL: https://issues.apache.org/jira/browse/BEAM-4831
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Reproduce by running
> {{./gradlew check}}
> {{Today this is failing.}}
> {{go vet seems to be trying to validate files in a vendor folder.}}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4831) Fix broken go vet task in the gradle build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4831:
---

Assignee: Robert Burke  (was: Luke Cwik)

> Fix broken go vet task in the gradle build
> --
>
> Key: BEAM-4831
> URL: https://issues.apache.org/jira/browse/BEAM-4831
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Reproduce by running
> {{./gradlew check}}
> {{Today this is failing.}}
> {{go vet seems to be trying to validate files in a vendor folder.}}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-4831] Ignore failures during :beam-sdks-go:vet

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit af61f5734f7c849d060609f544e13380a4663846
Merge: 9964774 6ebba9c
Author: Lukasz Cwik 
AuthorDate: Mon Jul 23 11:49:42 2018 -0700

[BEAM-4831] Ignore failures during :beam-sdks-go:vet

 sdks/go/build.gradle | 5 +
 1 file changed, 5 insertions(+)

[beam] branch master updated (9964774 -> af61f57)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9964774  Merge pull request #5998 from boyuanzz/continue_build
 add 6ebba9c  [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow 
./gradlew check to pass.
 new af61f57  [BEAM-4831] Ignore failures during :beam-sdks-go:vet

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/build.gradle | 5 +
 1 file changed, 5 insertions(+)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126225
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:46
Start Date: 23/Jul/18 18:46
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407161442
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126225)
Time Spent: 3h 20m  (was: 3h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126224
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:46
Start Date: 23/Jul/18 18:46
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407161442
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126224)
Time Spent: 3h 10m  (was: 3h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126226
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:46
Start Date: 23/Jul/18 18:46
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407161006
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126226)
Time Spent: 3.5h  (was: 3h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126223
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:45
Start Date: 23/Jul/18 18:45
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5935: [BEAM-4176] 
Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#issuecomment-407161088
 
 
   I don't have comments on this change yet, but wanted to just mention [this 
prototype](https://github.com/apache/beam/commit/b78f76f89f560406889dfa9c741e34d58bb05b7e)
 of an ArtifactStagingService that:
   - detects artifacts that have already been uploaded (by MD5)
   - closes the connection to prevent redundant artifacts from being uploaded
   - leaves the manifest with all the entries for each job (just with URIs that 
may point to where a previous job staged the artifact)
   
   It doesn't help the main pain point here (re-staging ~200MB of artifacts for 
each test case) because it doesn't work across artifact-server instances, but 
that's the next extension I'm planning to look into.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126223)
Time Spent: 2h 50m  (was: 2h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126222
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:45
Start Date: 23/Jul/18 18:45
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407160340
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126222)
Time Spent: 3h  (was: 2h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126221
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:45
Start Date: 23/Jul/18 18:45
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407161006
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126221)
Time Spent: 2h 50m  (was: 2h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=126220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126220
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:44
Start Date: 23/Jul/18 18:44
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6014: [BEAM-3709] 
Implementing new combine URNs in python.
URL: https://github.com/apache/beam/pull/6014#issuecomment-407160779
 
 
   Thanks, this LGTM.  Are there tests we can add at this stage?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126220)
Time Spent: 0.5h  (was: 20m)

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, sdk-py-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Python SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126219
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:42
Start Date: 23/Jul/18 18:42
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407143063
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126219)
Time Spent: 2h 40m  (was: 2.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=126218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126218
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:42
Start Date: 23/Jul/18 18:42
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407160340
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126218)
Time Spent: 2.5h  (was: 2h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126205
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204507807
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -23,14 +24,27 @@ applyJavaNature(
   },
 )
 
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure 
that
+ * the following projects are evaluated before we evaluate this project. This 
is because
+ * we are attempting to reference the "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":beam-runners-flink_2.11")
 
 Review comment:
   You reference `:beam-sdks-java-core` and `:beam-runners-core-java` 
sourceSets.test.output directly and not `:beam-runners-flink_2.11`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126205)
Time Spent: 2h  (was: 1h 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126212
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204510995
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import org.apache.beam.runners.reference.PortableRunner;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.hamcrest.Matchers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link TestPortableRunner} is a pipeline runner that wraps a {@link 
PortableRunner} when running
+ * tests against the {@link TestPipeline}.
+ *
+ * This runner requires a JobServerDriver with following methods.
 
 Review comment:
   Make the FlinkJobServerDriver implement a new interface 
`TestPortableRunner.JobServerDriver` which exposes these methods.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126212)
Time Spent: 2.5h  (was: 2h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126209
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204509017
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -81,6 +75,23 @@ private static void printUsage(CmdLineParser parser) {
 System.err.println();
   }
 
+  private static ServerConfiguration parseServerConfiguration(String[] args) {
+ServerConfiguration configuration = new ServerConfiguration();
 
 Review comment:
   nit: is there a reason why you want the contents of 
`parseServerConfiguration` to be outside of `fromParams`, the redirection 
reduces the readability of the code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126209)
Time Spent: 2h 20m  (was: 2h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126211
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204509792
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -109,28 +120,63 @@ private FlinkJobServerDriver(
   @Override
   public void run() {
 try {
-  GrpcFnServer server = createJobServer();
-  server.getServer().awaitTermination();
+  jobServer = createJobServer();
+  jobServer.getServer().awaitTermination();
 } catch (InterruptedException e) {
   LOG.warn("Job server interrupted", e);
 } catch (Exception e) {
   LOG.warn("Exception during job server creation", e);
+} finally {
+  stop();
+}
+  }
+
+  public String start() throws IOException {
+jobServer = createJobServer();
+return jobServer.getApiServiceDescriptor().getUrl();
+  }
+
+  public void stop() {
+if (jobServer != null) {
+  try {
+jobServer.close();
+LOG.info("JobServer stopped on {}", 
jobServer.getApiServiceDescriptor().getUrl());
+jobServer = null;
+  } catch (Exception e) {
+LOG.error("Error while closing the jobServer.");
+  }
+}
+if (artifactStagingServer != null) {
+  try {
+artifactStagingServer.close();
+LOG.info(
+"ArtifactStagingServer stopped on {}", 
jobServer.getApiServiceDescriptor().getUrl());
+artifactStagingServer = null;
+  } catch (Exception e) {
+LOG.error("Error while closing the artifactStagingServer.");
 
 Review comment:
   Log the error


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126211)
Time Spent: 2.5h  (was: 2h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126207
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204508001
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -23,14 +24,27 @@ applyJavaNature(
   },
 )
 
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure 
that
+ * the following projects are evaluated before we evaluate this project. This 
is because
+ * we are attempting to reference the "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":beam-runners-flink_2.11")
+
 description = "Apache Beam :: Runners :: Flink :: Job Server"
 
 apply plugin: "application"
 
 mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
 
+configurations {
+  validatesRunner
+}
+
 dependencies {
   compile project(path: ":beam-runners-flink_2.11", configuration: "shadow")
+  validatesRunner project(path: ":beam-runners-flink_2.11", configuration: 
"shadowTest")
 
 Review comment:
   You need to add `:beam-sdks-java-core / shadowTest` and 
`:beam-runners-core-java / shadowTest` as a `validatesRunner` dependency


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126207)
Time Spent: 2h 10m  (was: 2h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126204
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204505109
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -45,3 +59,55 @@ runShadow {
   // Enable remote debugging.
   jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
 }
+
+class PortableValidatesRunnerConfig {
+  String name
 
 Review comment:
   Please add comments for what each of these configuration properties do


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126204)
Time Spent: 1h 50m  (was: 1h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126206
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204509771
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -109,28 +120,63 @@ private FlinkJobServerDriver(
   @Override
   public void run() {
 try {
-  GrpcFnServer server = createJobServer();
-  server.getServer().awaitTermination();
+  jobServer = createJobServer();
+  jobServer.getServer().awaitTermination();
 } catch (InterruptedException e) {
   LOG.warn("Job server interrupted", e);
 } catch (Exception e) {
   LOG.warn("Exception during job server creation", e);
+} finally {
+  stop();
+}
+  }
+
+  public String start() throws IOException {
+jobServer = createJobServer();
+return jobServer.getApiServiceDescriptor().getUrl();
+  }
+
+  public void stop() {
+if (jobServer != null) {
+  try {
+jobServer.close();
+LOG.info("JobServer stopped on {}", 
jobServer.getApiServiceDescriptor().getUrl());
+jobServer = null;
+  } catch (Exception e) {
+LOG.error("Error while closing the jobServer.");
 
 Review comment:
   Log the error


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126206)
Time Spent: 2h 10m  (was: 2h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126208
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204511684
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import org.apache.beam.runners.reference.PortableRunner;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.hamcrest.Matchers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link TestPortableRunner} is a pipeline runner that wraps a {@link 
PortableRunner} when running
+ * tests against the {@link TestPipeline}.
+ *
+ * This runner requires a JobServerDriver with following methods.
+ *
+ * 
+ *   public static Object fromParams(String... params)
+ *   public String start() // Start JobServer and returns the JobServer 
host and port.
+ *   public void start() // Stop the JobServer and free all resources.
+ * 
+ *
+ * @see TestPipeline
+ */
+public class TestPortableRunner extends PipelineRunner {
+  private static final Logger LOG = 
LoggerFactory.getLogger(TestPortableRunner.class);
+  private static final ObjectMapper MAPPER =
+  new ObjectMapper()
+  
.registerModules(ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
+  private final PortablePipelineOptions options;
+
+  private TestPortableRunner(PortablePipelineOptions options) {
+this.options = options;
+  }
+
+  public static TestPortableRunner fromOptions(PipelineOptions options) {
+return new TestPortableRunner(options.as(PortablePipelineOptions.class));
+  }
+
+  @Override
+  public PipelineResult run(Pipeline pipeline) {
+TestPortablePipelineOptions testPortablePipelineOptions =
+options.as(TestPortablePipelineOptions.class);
+String jobServerHostPort;
+Object jobServerDriver;
+Class jobServerDriverClass;
+try {
+  jobServerDriverClass =
+  Class.forName(
+  testPortablePipelineOptions.getJobServerDriver(),
+  true,
+  ReflectHelpers.findClassLoader());
+  String[] parameters;
+  try {
+parameters =
+MAPPER.readValue(testPortablePipelineOptions.getJobServerConfig(), 
String[].class);
+  } catch (IOException e) {
+throw new IllegalArgumentException(
+String.format(
+"Arguments to jobServers should be a comma separated list of 
strings. Provided %s",
+testPortablePipelineOptions.getJobServerDriver()));
+  }
+  try {
+jobServerDriver =
+jobServerDriverClass
+.getMethod("fromParams", String[].class)
 
 Review comment:
   Reuse InstanceBuilder:
   
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/InstanceBuilder.java


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact 

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126210
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204510540
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
 ##
 @@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.Validation.Required;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+
+/** Options for {@link TestPortableRunner}. */
+public interface TestPortablePipelineOptions extends TestPipelineOptions, 
PortablePipelineOptions {
+
+  @Required
+  @Description(
+  "Fully qualified class name of TestJobServiceDriver capable of managing 
the JobService.")
+  String getJobServerDriver();
+
+  void setJobServerDriver(String jobServerDriver);
+
+  @Description("String containing comma separated arguments for the 
JobServer.")
+  String getJobServerConfig();
 
 Review comment:
   PipelineOptionsFactory supports parsing List or String[] 
automatically. There is no need to do this yourself.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126210)
Time Spent: 2.5h  (was: 2h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126203
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204504981
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -45,3 +59,55 @@ runShadow {
   // Enable remote debugging.
   jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
 }
+
+class PortableValidatesRunnerConfig {
+  String name
+  String jobServerDriver
+  String jobServerConfig
+  boolean streaming
+}
+
+def createPortableValidatesRunnerTask(Map m) {
+  def config = m as PortableValidatesRunnerConfig
+  tasks.create(name: config.name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+"--jobServerConfig=${config.jobServerConfig}",
+])
+classpath = configurations.validatesRunner
+testClassesDirs = 
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs, 
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
+maxParallelForks 1
+if (config.streaming) {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
+excludeCategories 'org.apache.beam.sdk.testing.UsesSchema'
+excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+  }
+} else {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.testing.UsesSchema'
+excludeCategories 
'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
+excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+  }
+}
+  }
+}
+
+createPortableValidatesRunnerTask(name: "validatesPortableRunner", 
jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver", 
jobServerConfig: "[]", streaming: false)
 
 Review comment:
   Do you expect `jobServerDriver` to change?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126203)
Time Spent: 1h 40m  (was: 1.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126202
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204504520
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -45,3 +59,55 @@ runShadow {
   // Enable remote debugging.
   jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
 }
+
+class PortableValidatesRunnerConfig {
+  String name
+  String jobServerDriver
+  String jobServerConfig
+  boolean streaming
+}
+
+def createPortableValidatesRunnerTask(Map m) {
 
 Review comment:
   nit: use a lambda and the implicit `it` parameter as it allows you to call 
the method without any paramemters or with parameters


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126202)
Time Spent: 1.5h  (was: 1h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126214
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204510246
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
 ##
 @@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.Validation.Required;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+
+/** Options for {@link TestPortableRunner}. */
+public interface TestPortablePipelineOptions extends TestPipelineOptions, 
PortablePipelineOptions {
+
+  @Required
+  @Description(
+  "Fully qualified class name of TestJobServiceDriver capable of managing 
the JobService.")
+  String getJobServerDriver();
 
 Review comment:
   This can be of type Class, PipelineOptionsFactory will convert string -> 
Class for you automatically during parsing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126214)
Time Spent: 2h 40m  (was: 2.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=126213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126213
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:38
Start Date: 23/Jul/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204509223
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java
 ##
 @@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import org.apache.beam.runners.reference.PortableRunner;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.hamcrest.Matchers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link TestPortableRunner} is a pipeline runner that wraps a {@link 
PortableRunner} when running
+ * tests against the {@link TestPipeline}.
+ *
+ * This runner requires a JobServerDriver with following methods.
+ *
+ * 
+ *   public static Object fromParams(String... params)
+ *   public String start() // Start JobServer and returns the JobServer 
host and port.
+ *   public void start() // Stop the JobServer and free all resources.
 
 Review comment:
   typo, start() -> stop()


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126213)
Time Spent: 2.5h  (was: 2h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #767

[Apache Jenkins Server]

See 


--
[...truncated 37.18 KB...]
at 
org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:42)
at 
org.gradle.api.internal.plugins.RuleBasedPluginTarget.applyImperative(RuleBasedPluginTarget.java:50)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:163)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.access$200(DefaultPluginManager.java:46)
at 
org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:251)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
at 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:143)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:128)
at org.gradle.api.plugins.GroovyPlugin.apply(GroovyPlugin.java:35)
at org.gradle.api.plugins.GroovyPlugin.apply(GroovyPlugin.java:31)
at 
org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:42)
at 
org.gradle.api.internal.plugins.RuleBasedPluginTarget.applyImperative(RuleBasedPluginTarget.java:50)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:163)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.access$200(DefaultPluginManager.java:46)
at 
org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:251)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
at 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:143)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:128)
at 
org.gradle.initialization.buildsrc.GroovyBuildSrcProjectConfigurationAction.execute(GroovyBuildSrcProjectConfigurationAction.java:27)
at 
org.gradle.initialization.buildsrc.GroovyBuildSrcProjectConfigurationAction.execute(GroovyBuildSrcProjectConfigurationAction.java:23)
at 
org.gradle.configuration.project.PluginsProjectConfigureActions.execute(PluginsProjectConfigureActions.java:45)
at 
org.gradle.configuration.project.PluginsProjectConfigureActions.execute(PluginsProjectConfigureActions.java:24)
at 
org.gradle.initialization.buildsrc.BuildSrcBuildListenerFactory$Listener.projectsLoaded(BuildSrcBuildListenerFactory.java:59)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:370)
at 
org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:352)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
at 
org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:340)
at 
org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:327)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
at 
org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:230)
at 

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126198
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:30
Start Date: 23/Jul/18 18:30
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #499: [BEAM-4802] 
Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407156626
 
 
   Looks like there is something wrong with presubmit tooling (CC: @jasonkuster)
   ```
   [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
/tmp/jenkins5733482887105358262.sh
   + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
409B6B1796C275462A1703113804BB82D39DC0E3 
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
   gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
   gpg: requesting key 39499BDB from hkp server keys.gnupg.net
   gpg: keyserver timed out
   gpg: keyserver receive failed: keyserver error
   Build step 'Execute shell' marked build as failure
   Setting status of 4ce9d1fa83bb84ba1533cba4c1431e10c1a8a8ec to FAILURE with 
url https://builds.apache.org/job/beam_PreCommit_Website_Stage/1224/ and 
message: 'FAILURE
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126198)
Time Spent: 2h 10m  (was: 2h)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4749) fastavro breaks macos tests

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553222#comment-16553222
 ] 

Ryan Williams commented on BEAM-4749:
-

That makes sense [~altay], thanks! I filed 
[fastavro#253|https://github.com/fastavro/fastavro/issues/253] about it

> fastavro breaks macos tests
> ---
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in 
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>  
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: 
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: 
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
>  '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 
> 'install', 
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>  
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> 

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126194
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:24
Start Date: 23/Jul/18 18:24
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #499: [BEAM-4802] 
Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407154652
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126194)
Time Spent: 2h  (was: 1h 50m)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #801

[Apache Jenkins Server]

See 


--
[...truncated 36.54 KB...]
at 
org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:42)
at 
org.gradle.api.internal.plugins.RuleBasedPluginTarget.applyImperative(RuleBasedPluginTarget.java:50)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:163)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.access$200(DefaultPluginManager.java:46)
at 
org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:251)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
at 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:143)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:128)
at org.gradle.api.plugins.GroovyPlugin.apply(GroovyPlugin.java:35)
at org.gradle.api.plugins.GroovyPlugin.apply(GroovyPlugin.java:31)
at 
org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:42)
at 
org.gradle.api.internal.plugins.RuleBasedPluginTarget.applyImperative(RuleBasedPluginTarget.java:50)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:163)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.access$200(DefaultPluginManager.java:46)
at 
org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:251)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
at 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:143)
at 
org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:128)
at 
org.gradle.initialization.buildsrc.GroovyBuildSrcProjectConfigurationAction.execute(GroovyBuildSrcProjectConfigurationAction.java:27)
at 
org.gradle.initialization.buildsrc.GroovyBuildSrcProjectConfigurationAction.execute(GroovyBuildSrcProjectConfigurationAction.java:23)
at 
org.gradle.configuration.project.PluginsProjectConfigureActions.execute(PluginsProjectConfigureActions.java:45)
at 
org.gradle.configuration.project.PluginsProjectConfigureActions.execute(PluginsProjectConfigureActions.java:24)
at 
org.gradle.initialization.buildsrc.BuildSrcBuildListenerFactory$Listener.projectsLoaded(BuildSrcBuildListenerFactory.java:59)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:370)
at 
org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:352)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
at 
org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:340)
at 
org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:327)
at 
org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
at 
org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:230)
at 

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126193
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:23
Start Date: 23/Jul/18 18:23
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #499: [BEAM-4802] 
Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407154610
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126193)
Time Spent: 1h 50m  (was: 1h 40m)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #460

[Apache Jenkins Server]

See 


--
[...truncated 29.81 KB...]
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.assembleParameters(DefaultServiceRegistry.java:780)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.assembleParameters(DefaultServiceRegistry.java:780)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.assembleParameters(DefaultServiceRegistry.java:780)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.assembleParameters(DefaultServiceRegistry.java:780)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.assembleParameters(DefaultServiceRegistry.java:780)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:766)
at 
org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectServiceProvider.getInstance(DefaultServiceRegistry.java:571)
at 
org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:628)
at 
org.gradle.internal.service.DefaultServiceRegistry.find(DefaultServiceRegistry.java:295)
at 
org.gradle.internal.service.DefaultServiceRegistry.get(DefaultServiceRegistry.java:284)
at 
org.gradle.internal.service.DefaultServiceRegistry.get(DefaultServiceRegistry.java:279)
at 
org.gradle.internal.service.scopes.BuildScopeServices.defaultScriptPluginFactory(BuildScopeServices.java:306)
at 
org.gradle.internal.service.scopes.BuildScopeServices.createScriptPluginFactory(BuildScopeServices.java:298)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
at 
org.gradle.internal.service.ReflectionBasedServiceMethod.invoke(ReflectionBasedServiceMethod.java:35)
at 
org.gradle.internal.service.DefaultServiceRegistry$FactoryMethodService.invokeMethod(DefaultServiceRegistry.java:814)
... 51 more
Caused by: org.gradle.cache.LockTimeoutException: Timeout waiting to lock file 
hash cache (/home/jenkins/.gradle/caches/4.8/fileHashes). It is currently in 
use by another Gradle instance.
Owner PID: 811
Our PID: 15044
Owner Operation: 
Our operation: 
Lock file: /home/jenkins/.gradle/caches/4.8/fileHashes/fileHashes.lock
at 
org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.lock(DefaultFileLockManager.java:284)
at 
org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.(DefaultFileLockManager.java:154)
at 
org.gradle.cache.internal.DefaultFileLockManager.lock(DefaultFileLockManager.java:100)
at 
org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess.incrementLockCount(LockOnDemandCrossProcessCacheAccess.java:105)
at 
org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess.acquireFileLock(LockOnDemandCrossProcessCacheAccess.java:161)
at 
org.gradle.cache.internal.DefaultCacheAccess.onStartWork(DefaultCacheAccess.java:368)
at 

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126184
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:16
Start Date: 23/Jul/18 18:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #499: 
[BEAM-4802] Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#discussion_r204498276
 
 

 ##
 File path: src/contribute/dependencies.md
 ##
 @@ -44,14 +44,25 @@ One common solution for the diamond dependency problem is 
[semantic versioning](
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list. In the future we hope to automatically create JIRAs 
based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 * A minor release of a dependency due to a critical security vulnerability. 
 * A dependency conflict that was was triggered by a minor version release of a 
Beam dependency (this does not apply to Java SDK that depends on exact minor 
versions of dependencies).
 
 These kind of urgently required upgrades might not get automatically picked up 
by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+## JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be created 
per significant outdated dependency base on the report. A bot named *Beam Jira 
Bot* was created for managing JIRA issues. Beam community agrees on the 
following policies that creates and updates issues.
+* Issues will be named as "Beam Dependency Update Request:  
".
+* Issues will be created under the component *"dependencies"*
+* Issues will be assigned to the primary owner of the dependencies, who are 
mentioned in the dependency ownership files. ([Java Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml)
 and [Python Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml))
+* If more than one owners found for a dependency, the first owner will be 
picked as the primary owner, the others will be pinged in the issue's 
description.
+* If no owners found, leave the assignee empty. The component lead is 
responsible for triage the issue.
+* Avoid creating duplicate issues. Updating the descriptions of the open 
issues created by previous dependency updates inspection.
 
 Review comment:
   Means the previous 'dependency check'. I changed the phrase. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126184)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126185
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:16
Start Date: 23/Jul/18 18:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #499: 
[BEAM-4802] Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#discussion_r204498304
 
 

 ##
 File path: src/contribute/dependencies.md
 ##
 @@ -44,14 +44,25 @@ One common solution for the diamond dependency problem is 
[semantic versioning](
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list. In the future we hope to automatically create JIRAs 
based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 * A minor release of a dependency due to a critical security vulnerability. 
 * A dependency conflict that was was triggered by a minor version release of a 
Beam dependency (this does not apply to Java SDK that depends on exact minor 
versions of dependencies).
 
 These kind of urgently required upgrades might not get automatically picked up 
by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+## JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be created 
per significant outdated dependency base on the report. A bot named *Beam Jira 
Bot* was created for managing JIRA issues. Beam community agrees on the 
following policies that creates and updates issues.
+* Issues will be named as "Beam Dependency Update Request:  
".
+* Issues will be created under the component *"dependencies"*
+* Issues will be assigned to the primary owner of the dependencies, who are 
mentioned in the dependency ownership files. ([Java Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml)
 and [Python Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml))
+* If more than one owners found for a dependency, the first owner will be 
picked as the primary owner, the others will be pinged in the issue's 
description.
+* If no owners found, leave the assignee empty. The component lead is 
responsible for triage the issue.
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126185)
Time Spent: 1h 40m  (was: 1.5h)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126183
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 23/Jul/18 18:16
Start Date: 23/Jul/18 18:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #499: 
[BEAM-4802] Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#discussion_r204498350
 
 

 ##
 File path: src/contribute/dependencies.md
 ##
 @@ -44,14 +44,25 @@ One common solution for the diamond dependency problem is 
[semantic versioning](
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list. In the future we hope to automatically create JIRAs 
based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 * A minor release of a dependency due to a critical security vulnerability. 
 * A dependency conflict that was was triggered by a minor version release of a 
Beam dependency (this does not apply to Java SDK that depends on exact minor 
versions of dependencies).
 
 These kind of urgently required upgrades might not get automatically picked up 
by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+## JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be created 
per significant outdated dependency base on the report. A bot named *Beam Jira 
Bot* was created for managing JIRA issues. Beam community agrees on the 
following policies that creates and updates issues.
 
 Review comment:
   done. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126183)
Time Spent: 1.5h  (was: 1h 20m)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)