Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3192

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3191

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2346) Failure in BigQueryTornadoesIT postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020612#comment-16020612
 ] 

ASF GitHub Bot commented on BEAM-2346:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3206

[BEAM-2346] Revert "Include Additional PTransform inputs in Transform Nodes"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This reverts commit 9336230d2a5c18bae89908bcd60db8ea96b7906d.

It appears to break `BigQueryTornadoesIT`: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/3884/

I have confirmed that the failure is eliminated by this revert.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam revert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit ee92a837b899a83102831b2af22a0b8d8deeea9b
Author: Kenneth Knowles 
Date:   2017-05-23T03:27:52Z

Revert "Include Additional PTransform inputs in Transform Nodes"

This reverts commit 9336230d2a5c18bae89908bcd60db8ea96b7906d.

It appears to break the BigQueryTornadoesIT.




> Failure in BigQueryTornadoesIT postcommit
> -
>
> Key: BEAM-2346
> URL: https://issues.apache.org/jira/browse/BEAM-2346
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> Failure began here: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/3884/
> Because of: https://github.com/apache/beam/pull/3188



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3206: [BEAM-2346] Revert "Include Additional PTransform i...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3206

[BEAM-2346] Revert "Include Additional PTransform inputs in Transform Nodes"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This reverts commit 9336230d2a5c18bae89908bcd60db8ea96b7906d.

It appears to break `BigQueryTornadoesIT`: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/3884/

I have confirmed that the failure is eliminated by this revert.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam revert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit ee92a837b899a83102831b2af22a0b8d8deeea9b
Author: Kenneth Knowles 
Date:   2017-05-23T03:27:52Z

Revert "Include Additional PTransform inputs in Transform Nodes"

This reverts commit 9336230d2a5c18bae89908bcd60db8ea96b7906d.

It appears to break the BigQueryTornadoesIT.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2346) Failure in BigQueryTornadoesIT postcommit

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2346:
-

 Summary: Failure in BigQueryTornadoesIT postcommit
 Key: BEAM-2346
 URL: https://issues.apache.org/jira/browse/BEAM-2346
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker


Failure began here: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/3884/

Because of: https://github.com/apache/beam/pull/3188



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2145

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3887

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1538

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #24

[Apache Jenkins Server]

See 


Changes:

[klk] Fix compile error from bad merge

--
[...truncated 1.09 MB...]
at 
com.google.cloud.dataflow.worker.runners.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
at 
com.google.cloud.dataflow.worker.runners.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:74)
at 
com.google.cloud.dataflow.worker.runners.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:101)
at 
com.google.cloud.dataflow.worker.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:198)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:341)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:297)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:244)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:125)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:105)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:92)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: java.io.IOException: 
Failed closing channel to 
gs://temp-storage-for-end-to-end-tests/WindowedWordCountIT.testWindowedWordCountInBatchStaticSharding-2017-05-23T02:27:17.039+-197020975/output/.temp-beam-2017-05-143_02-27-20-3/d4a63808-5c17-4fca-9de1-363f7c522dce
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.io.WriteFiles$WriteShardedBundles$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:197)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:155)
at 
com.google.cloud.dataflow.worker.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:246)
at 
com.google.cloud.dataflow.worker.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at 
com.google.cloud.dataflow.worker.runners.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:144)
... 23 more
Caused by: java.io.IOException: Failed closing channel to 
gs://temp-storage-for-end-to-end-tests/WindowedWordCountIT.testWindowedWordCountInBatchStaticSharding-2017-05-23T02:27:17.039+-197020975/output/.temp-beam-2017-05-143_02-27-20-3/d4a63808-5c17-4fca-9de1-363f7c522dce
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.close(FileBasedSink.java:938)
at 
org.apache.beam.sdk.io.WriteFiles$WriteShardedBundles.processElement(WriteFiles.java:372)
Caused by: java.io.IOException: 
com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
  "code" : 503,
  "errors" : [ {
"domain" : "global",
"message" : "Backend Error",
"reason" : "backendError"
  } ],
  "message" : "Backend Error"
}
at 
com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:431)
at 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2144

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3884

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3190

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #24

[Apache Jenkins Server]

See 


Changes:

[klk] Fix compile error from bad merge

--
[...truncated 1.05 MB...]
2017-05-23\T\02:23:55.703 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\02:23:55.703 [INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\02:23:55.703 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:23:55.703 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:23:55.704 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.705 [INFO] Excluding 
io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:23:55.706 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\02:23:55.707 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\02:23:55.708 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-05-23\T\02:23:55.709 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #24

[Apache Jenkins Server]

See 


Changes:

[klk] Fix compile error from bad merge

--
[...truncated 1.06 MB...]
2017-05-23\T\02:21:47.378 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-05-23\T\02:21:47.378 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2290

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #2903

[Apache Jenkins Server]

See 

beam git commit: Fix compile error from bad merge

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master a32bef96d -> 9da46fd05


Fix compile error from bad merge


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9da46fd0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9da46fd0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9da46fd0

Branch: refs/heads/master
Commit: 9da46fd053ef3e7538733d45cacad435c312e51b
Parents: a32bef9
Author: Kenneth Knowles 
Authored: Mon May 22 19:02:56 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon May 22 19:02:56 2017 -0700

--
 .../beam/runners/core/construction/WindowIntoTranslatorTest.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9da46fd0/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslatorTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslatorTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslatorTest.java
index fbac565..eaefe2e 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslatorTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslatorTest.java
@@ -84,7 +84,8 @@ public class WindowIntoTranslatorTest {
   public void visitPrimitiveTransform(Node node) {
 if (node.getTransform() instanceof Window.Assign) {
   checkState(assign.get() == null);
-  assign.set((AppliedPTransform>) 
node.toAppliedPTransform());
+  assign.set(
+  (AppliedPTransform>) 
node.toAppliedPTransform(getPipeline()));
 }
   }
 });

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1537

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

[klk] Add WindowIntoTranslator

--
[...truncated 394.81 KB...]
2017-05-23T01:59:46.564 [INFO] Running 
org.apache.beam.sdk.transforms.windowing.NeverTest
2017-05-23T01:59:46.566 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.transforms.windowing.NeverTest
2017-05-23T01:59:46.569 [INFO] Running org.apache.beam.sdk.transforms.ValuesTest
2017-05-23T01:59:46.571 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.transforms.ValuesTest
2017-05-23T01:59:46.572 [INFO] Running 
org.apache.beam.sdk.transforms.PartitionTest
2017-05-23T01:59:46.583 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.009 s - in org.apache.beam.sdk.transforms.PartitionTest
2017-05-23T01:59:46.586 [INFO] Running org.apache.beam.sdk.transforms.MaxTest
2017-05-23T01:59:46.588 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.transforms.MaxTest
2017-05-23T01:59:46.588 [INFO] Running org.apache.beam.sdk.transforms.ParDoTest
2017-05-23T01:59:46.671 [WARNING] Tests run: 14, Failures: 0, Errors: 0, 
Skipped: 1, Time elapsed: 0.08 s - in org.apache.beam.sdk.transforms.ParDoTest
2017-05-23T01:59:46.671 [INFO] Running org.apache.beam.sdk.transforms.CreateTest
2017-05-23T01:59:47.010 [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.336 s - in org.apache.beam.sdk.transforms.CreateTest
2017-05-23T01:59:47.010 [INFO] Running 
org.apache.beam.sdk.transforms.CombineTest
2017-05-23T01:59:47.025 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.013 s - in org.apache.beam.sdk.transforms.CombineTest
2017-05-23T01:59:47.025 [INFO] Running org.apache.beam.sdk.util.VarIntTest
2017-05-23T01:59:47.033 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.006 s - in org.apache.beam.sdk.util.VarIntTest
2017-05-23T01:59:47.033 [INFO] Running org.apache.beam.sdk.util.StreamUtilsTest
2017-05-23T01:59:47.040 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.005 s - in org.apache.beam.sdk.util.StreamUtilsTest
2017-05-23T01:59:47.040 [INFO] Running org.apache.beam.sdk.util.StringUtilsTest
2017-05-23T01:59:47.043 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.001 s - in org.apache.beam.sdk.util.StringUtilsTest
2017-05-23T01:59:47.043 [INFO] Running 
org.apache.beam.sdk.util.MutationDetectorsTest
2017-05-23T01:59:47.053 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.008 s - in org.apache.beam.sdk.util.MutationDetectorsTest
2017-05-23T01:59:47.053 [INFO] Running 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
2017-05-23T01:59:47.059 [INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.004 s - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
2017-05-23T01:59:47.059 [INFO] Running org.apache.beam.sdk.util.ZipFilesTest
2017-05-23T01:59:47.080 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.019 s - in org.apache.beam.sdk.util.ZipFilesTest
2017-05-23T01:59:47.080 [INFO] Running org.apache.beam.sdk.util.ApiSurfaceTest
2017-05-23T01:59:47.093 [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.011 s - in org.apache.beam.sdk.util.ApiSurfaceTest
2017-05-23T01:59:47.093 [INFO] Running 
org.apache.beam.sdk.util.UnownedInputStreamTest
2017-05-23T01:59:47.095 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.UnownedInputStreamTest
2017-05-23T01:59:47.096 [INFO] Running org.apache.beam.sdk.util.ReleaseInfoTest
2017-05-23T01:59:47.098 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.ReleaseInfoTest
2017-05-23T01:59:47.098 [INFO] Running 
org.apache.beam.sdk.util.WindowedValueTest
2017-05-23T01:59:47.100 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.WindowedValueTest
2017-05-23T01:59:47.100 [INFO] Running 
org.apache.beam.sdk.util.common.ReflectHelpersTest
2017-05-23T01:59:47.105 [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.003 s - in org.apache.beam.sdk.util.common.ReflectHelpersTest
2017-05-23T01:59:47.105 [INFO] Running 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
2017-05-23T01:59:47.107 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
2017-05-23T01:59:47.107 [INFO] Running 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
2017-05-23T01:59:47.388 [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.279 s - in 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #23

[Apache Jenkins Server]

See 


--
[...truncated 438.68 KB...]
2017-05-23\T\02:00:55.463 [INFO] Starting audit...
Audit done.
2017-05-23\T\02:00:55.464 [INFO] 
2017-05-23\T\02:00:55.464 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.467 [INFO] 
2017-05-23\T\02:00:55.467 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.467 [INFO] 
2017-05-23\T\02:00:55.467 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.470 [INFO] Building jar: 

2017-05-23\T\02:00:55.475 [INFO] 
2017-05-23\T\02:00:55.475 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.476 [INFO] 
2017-05-23\T\02:00:55.476 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.481 [INFO] Building jar: 

2017-05-23\T\02:00:55.488 [INFO] 
2017-05-23\T\02:00:55.488 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\02:00:55.492 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\02:00:58.045 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\02:00:58.046 [INFO] Replacing 

 with 

2017-05-23\T\02:00:58.046 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\02:00:58.046 [INFO] Replacing 

 with 

2017-05-23\T\02:00:58.046 [INFO] Dependency-reduced POM written at: 

2017-05-23\T\02:00:58.159 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #23

[Apache Jenkins Server]

See 


--
[...truncated 438.72 KB...]
2017-05-23\T\01:59:37.762 [INFO] Starting audit...
Audit done.
2017-05-23\T\01:59:37.763 [INFO] 
2017-05-23\T\01:59:37.763 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.767 [INFO] 
2017-05-23\T\01:59:37.767 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.767 [INFO] 
2017-05-23\T\01:59:37.767 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.770 [INFO] Building jar: 

2017-05-23\T\01:59:37.777 [INFO] 
2017-05-23\T\01:59:37.777 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.778 [INFO] 
2017-05-23\T\01:59:37.778 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.781 [INFO] Building jar: 

2017-05-23\T\01:59:37.788 [INFO] 
2017-05-23\T\01:59:37.789 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\01:59:37.791 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\01:59:39.669 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\01:59:39.669 [INFO] Replacing 

 with 

2017-05-23\T\01:59:39.669 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\01:59:39.669 [INFO] Replacing 

 with 

2017-05-23\T\01:59:39.670 [INFO] Dependency-reduced POM written at: 

2017-05-23\T\01:59:39.730 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #23

[Apache Jenkins Server]

See 


--
[...truncated 437.60 KB...]
2017-05-23\T\01:58:30.459 [INFO] Starting audit...
Audit done.
2017-05-23\T\01:58:30.459 [INFO] 
2017-05-23\T\01:58:30.459 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.462 [INFO] 
2017-05-23\T\01:58:30.462 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.463 [INFO] 
2017-05-23\T\01:58:30.463 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.466 [INFO] Building jar: 

2017-05-23\T\01:58:30.471 [INFO] 
2017-05-23\T\01:58:30.471 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.472 [INFO] 
2017-05-23\T\01:58:30.472 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.475 [INFO] Building jar: 

2017-05-23\T\01:58:30.483 [INFO] 
2017-05-23\T\01:58:30.483 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\01:58:30.486 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\01:58:33.269 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\01:58:33.269 [INFO] Replacing 

 with 

2017-05-23\T\01:58:33.269 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\01:58:33.269 [INFO] Replacing 

 with 

2017-05-23\T\01:58:33.269 [INFO] Dependency-reduced POM written at: 

2017-05-23\T\01:58:33.319 [INFO] 
2017-05-23\T\01:58:33.319 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3886

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

--
[...truncated 505.58 KB...]
2017-05-23T01:56:04.388 [INFO] Exclude: **/.idea/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/*.egg-info/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/package-list
2017-05-23T01:56:04.388 [INFO] Exclude: **/user.avsc
2017-05-23T01:56:04.388 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-23T01:56:04.388 [INFO] Exclude: **/test/**/.placeholder
2017-05-23T01:56:04.388 [INFO] Exclude: .repository/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/nose-*.egg/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/.eggs/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/.tox/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: **/.checkstyle
2017-05-23T01:56:04.388 [INFO] Exclude: **/.classpath
2017-05-23T01:56:04.388 [INFO] Exclude: **/.factorypath
2017-05-23T01:56:04.388 [INFO] Exclude: **/.project
2017-05-23T01:56:04.388 [INFO] Exclude: **/.settings/**/*
2017-05-23T01:56:04.388 [INFO] Exclude: 
**/sdks/python/apache_beam/runners/api/*.py
2017-05-23T01:56:04.413 [INFO] 647 resources included (use -debug for more 
details)
2017-05-23T01:56:04.830 [INFO] Rat check: Summary over all files. Unapproved: 
0, unknown: 0, generated: 0, approved: 647 licenses.
2017-05-23T01:56:04.880 [INFO] 
2017-05-23T01:56:04.880 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-runners-parent ---
2017-05-23T01:56:04.882 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-05-23T01:56:05.903 [INFO]  
   
2017-05-23T01:56:05.903 [INFO] 

2017-05-23T01:56:05.903 [INFO] Building Apache Beam :: Runners :: Core 
Construction Java 2.1.0-SNAPSHOT
2017-05-23T01:56:05.903 [INFO] 

2017-05-23T01:56:05.917 [INFO] 
2017-05-23T01:56:05.917 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-core-construction-java ---
2017-05-23T01:56:05.920 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-23T01:56:05.972 [INFO] 
2017-05-23T01:56:05.972 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-core-construction-java ---
2017-05-23T01:56:06.380 [INFO] 
2017-05-23T01:56:06.380 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-core-construction-java ---
2017-05-23T01:56:06.439 [INFO] 
2017-05-23T01:56:06.439 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-core-construction-java ---
2017-05-23T01:56:06.688 [INFO] 
2017-05-23T01:56:06.688 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-core-construction-java ---
2017-05-23T01:56:06.690 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-23T01:56:06.690 [INFO] skip non existing resourceDirectory 

2017-05-23T01:56:06.690 [INFO] Copying 3 resources
2017-05-23T01:56:06.741 [INFO] 
2017-05-23T01:56:06.741 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-core-construction-java ---
2017-05-23T01:56:06.751 [INFO] Changes detected - recompiling the module!
2017-05-23T01:56:06.751 [INFO] Compiling 32 source files to 

2017-05-23T01:56:07.468 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-23T01:56:07.468 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-23T01:56:07.468 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-05-23T01:56:07.468 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-05-23T01:56:07.468 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2143

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

--
[...truncated 405.07 KB...]
2017-05-23T01:55:22.910 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23T01:55:22.910 [INFO] Including net.bytebuddy:byte-buddy:jar:1.6.8 in 
the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Including 
org.apache.commons:commons-compress:jar:1.9 in the shaded jar.
2017-05-23T01:55:22.910 [INFO] Including 
org.apache.commons:commons-lang3:jar:3.5 in the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2017-05-23T01:55:22.910 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-05-23T01:55:22.910 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-05-23T01:55:32.388 [INFO] Replacing original artifact with shaded artifact.
2017-05-23T01:55:32.388 [INFO] Replacing 

 with 

2017-05-23T01:55:32.388 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23T01:55:32.388 [INFO] Replacing 

 with 

2017-05-23T01:55:32.388 [INFO] Dependency-reduced POM written at: 

2017-05-23T01:55:32.611 [INFO] 
2017-05-23T01:55:32.612 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-core ---
2017-05-23T01:55:32.846 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2017-05-23T01:55:41.942 [INFO]  
   
2017-05-23T01:55:41.942 [INFO] 

2017-05-23T01:55:41.942 [INFO] Building Apache Beam :: SDKs :: Java :: IO 
2.1.0-SNAPSHOT
2017-05-23T01:55:41.942 [INFO] 

2017-05-23T01:55:41.945 [INFO] 
2017-05-23T01:55:41.945 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-io-parent ---
2017-05-23T01:55:41.947 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-23T01:55:42.011 [INFO] 
2017-05-23T01:55:42.011 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-sdks-java-io-parent ---
2017-05-23T01:55:42.132 [INFO] 
2017-05-23T01:55:42.132 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-io-parent ---
2017-05-23T01:55:42.184 [INFO] 
2017-05-23T01:55:42.184 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-io-parent ---
2017-05-23T01:55:42.241 [INFO] 
2017-05-23T01:55:42.241 [INFO] --- maven-checkstyle-plugin:2.17:check (default) 
@ beam-sdks-java-io-parent ---
2017-05-23T01:55:42.706 [INFO] Starting audit...
Audit done.
2017-05-23T01:55:42.757 [INFO] 
2017-05-23T01:55:42.757 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-parent ---
2017-05-23T01:55:42.828 [INFO] 
2017-05-23T01:55:42.828 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-parent ---
2017-05-23T01:55:43.000 [INFO] 
2017-05-23T01:55:43.000 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #22

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

[klk] Add WindowIntoTranslator

--
[...truncated 437.61 KB...]
2017-05-23\T\01:54:21.023 [INFO] Starting audit...
Audit done.
2017-05-23\T\01:54:21.023 [INFO] 
2017-05-23\T\01:54:21.023 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.027 [INFO] 
2017-05-23\T\01:54:21.027 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.027 [INFO] 
2017-05-23\T\01:54:21.027 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.030 [INFO] Building jar: 

2017-05-23\T\01:54:21.040 [INFO] 
2017-05-23\T\01:54:21.040 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.041 [INFO] 
2017-05-23\T\01:54:21.041 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.046 [INFO] Building jar: 

2017-05-23\T\01:54:21.053 [INFO] 
2017-05-23\T\01:54:21.053 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\01:54:21.057 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\01:54:24.075 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\01:54:24.075 [INFO] Replacing 

 with 

2017-05-23\T\01:54:24.076 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\01:54:24.076 [INFO] Replacing 

 with 

2017-05-23\T\01:54:24.076 [INFO] Dependency-reduced POM written at: 

2017-05-23\T\01:54:24.141 [INFO] 
2017-05-23\T\01:54:24.141 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2902

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

--
[...truncated 404.17 KB...]
2017-05-23T01:52:05.758 [INFO] Running 
org.apache.beam.sdk.util.NumberedShardedFileTest
2017-05-23T01:52:05.806 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.046 s - in org.apache.beam.sdk.util.NumberedShardedFileTest
2017-05-23T01:52:05.806 [INFO] Running 
org.apache.beam.sdk.util.UnownedOutputStreamTest
2017-05-23T01:52:05.810 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.002 s - in org.apache.beam.sdk.util.UnownedOutputStreamTest
2017-05-23T01:52:05.810 [INFO] Running 
org.apache.beam.sdk.util.CombineFnUtilTest
2017-05-23T01:52:05.824 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.012 s - in org.apache.beam.sdk.util.CombineFnUtilTest
2017-05-23T01:52:05.824 [INFO] Running 
org.apache.beam.sdk.util.BucketingFunctionTest
2017-05-23T01:52:05.829 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.003 s - in org.apache.beam.sdk.util.BucketingFunctionTest
2017-05-23T01:52:05.830 [INFO] Running org.apache.beam.sdk.util.NameUtilsTest
2017-05-23T01:52:05.837 [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.005 s - in org.apache.beam.sdk.util.NameUtilsTest
2017-05-23T01:52:05.838 [INFO] Running 
org.apache.beam.sdk.util.UserCodeExceptionTest
2017-05-23T01:52:05.841 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.UserCodeExceptionTest
2017-05-23T01:52:05.844 [INFO] Running 
org.apache.beam.sdk.util.MovingFunctionTest
2017-05-23T01:52:05.846 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.MovingFunctionTest
2017-05-23T01:52:06.338 [INFO] 
2017-05-23T01:52:06.338 [INFO] Results:
2017-05-23T01:52:06.338 [INFO] 
2017-05-23T01:52:06.338 [WARNING] Tests run: 2311, Failures: 0, Errors: 0, 
Skipped: 1
2017-05-23T01:52:06.338 [INFO] 
[JENKINS] Recording test results
2017-05-23T01:52:35.861 [INFO] 
2017-05-23T01:52:35.861 [INFO] --- jacoco-maven-plugin:0.7.8:report (report) @ 
beam-sdks-java-core ---
2017-05-23T01:52:35.961 [INFO] Loading execution data file 

2017-05-23T01:52:36.329 [INFO] Analyzed bundle 'Apache Beam :: SDKs :: Java :: 
Core' with 1037 classes
2017-05-23T01:52:37.910 [INFO] 
2017-05-23T01:52:37.910 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
2017-05-23T01:52:38.013 [INFO] 
2017-05-23T01:52:38.013 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-05-23T01:52:38.212 [INFO] Building jar: 

2017-05-23T01:52:38.434 [INFO] 
2017-05-23T01:52:38.434 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-05-23T01:52:38.535 [INFO] 
2017-05-23T01:52:38.535 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-05-23T01:52:38.725 [INFO] Building jar: 

2017-05-23T01:52:38.916 [INFO] 
2017-05-23T01:52:38.916 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-05-23T01:52:38.923 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2017-05-23T01:52:38.924 [INFO] Including 
com.google.protobuf:protobuf-java:jar:3.2.0 in the shaded jar.
2017-05-23T01:52:38.924 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23T01:52:38.924 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23T01:52:38.924 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23T01:52:38.925 [INFO] Including net.bytebuddy:byte-buddy:jar:1.6.8 in 
the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:52:38.925 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23T01:52:38.925 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #22

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

[klk] Add WindowIntoTranslator

--
[...truncated 439.19 KB...]
2017-05-23\T\01:52:16.193 [INFO] Starting audit...
Audit done.
2017-05-23\T\01:52:16.194 [INFO] 
2017-05-23\T\01:52:16.194 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.197 [INFO] 
2017-05-23\T\01:52:16.197 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.197 [INFO] 
2017-05-23\T\01:52:16.197 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.199 [INFO] Building jar: 

2017-05-23\T\01:52:16.206 [INFO] 
2017-05-23\T\01:52:16.206 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.207 [INFO] 
2017-05-23\T\01:52:16.207 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.211 [INFO] Building jar: 

2017-05-23\T\01:52:16.218 [INFO] 
2017-05-23\T\01:52:16.218 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\01:52:16.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\01:52:16.220 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\01:52:16.221 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\01:52:18.560 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\01:52:18.560 [INFO] Replacing 

 with 

2017-05-23\T\01:52:18.560 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\01:52:18.560 [INFO] Replacing 

 with 

2017-05-23\T\01:52:18.561 [INFO] Dependency-reduced POM written at: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #22

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ReadTranslator

[klk] Add WindowIntoTranslator

--
[...truncated 438.76 KB...]
2017-05-23\T\01:50:40.798 [INFO] Starting audit...
Audit done.
2017-05-23\T\01:50:40.799 [INFO] 
2017-05-23\T\01:50:40.799 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.802 [INFO] 
2017-05-23\T\01:50:40.802 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.802 [INFO] 
2017-05-23\T\01:50:40.802 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.804 [INFO] Building jar: 

2017-05-23\T\01:50:40.809 [INFO] 
2017-05-23\T\01:50:40.809 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.810 [INFO] 
2017-05-23\T\01:50:40.810 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.813 [INFO] Building jar: 

2017-05-23\T\01:50:40.819 [INFO] 
2017-05-23\T\01:50:40.819 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-io-common ---
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.9 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.5 from the shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23\T\01:50:40.821 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-05-23\T\01:50:42.696 [INFO] Replacing original artifact with shaded 
artifact.
2017-05-23\T\01:50:42.696 [INFO] Replacing 

 with 

2017-05-23\T\01:50:42.696 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23\T\01:50:42.696 [INFO] Replacing 

 with 

2017-05-23\T\01:50:42.697 [INFO] Dependency-reduced POM written at: 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3885

[Apache Jenkins Server]

See 


Changes:

[klk] Add WindowIntoTranslator

--
[...truncated 509.41 KB...]
2017-05-23T01:47:25.465 [INFO] Exclude: **/.idea/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/*.egg-info/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/package-list
2017-05-23T01:47:25.465 [INFO] Exclude: **/user.avsc
2017-05-23T01:47:25.465 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-23T01:47:25.465 [INFO] Exclude: **/test/**/.placeholder
2017-05-23T01:47:25.465 [INFO] Exclude: .repository/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/nose-*.egg/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/.eggs/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/.tox/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: **/.checkstyle
2017-05-23T01:47:25.465 [INFO] Exclude: **/.classpath
2017-05-23T01:47:25.465 [INFO] Exclude: **/.factorypath
2017-05-23T01:47:25.465 [INFO] Exclude: **/.project
2017-05-23T01:47:25.465 [INFO] Exclude: **/.settings/**/*
2017-05-23T01:47:25.465 [INFO] Exclude: 
**/sdks/python/apache_beam/runners/api/*.py
2017-05-23T01:47:25.477 [INFO] 645 resources included (use -debug for more 
details)
2017-05-23T01:47:25.694 [INFO] Rat check: Summary over all files. Unapproved: 
0, unknown: 0, generated: 0, approved: 645 licenses.
2017-05-23T01:47:25.744 [INFO] 
2017-05-23T01:47:25.744 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-runners-parent ---
2017-05-23T01:47:25.746 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-05-23T01:47:26.574 [INFO]  
   
2017-05-23T01:47:26.574 [INFO] 

2017-05-23T01:47:26.574 [INFO] Building Apache Beam :: Runners :: Core 
Construction Java 2.1.0-SNAPSHOT
2017-05-23T01:47:26.574 [INFO] 

2017-05-23T01:47:26.580 [INFO] 
2017-05-23T01:47:26.580 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-core-construction-java ---
2017-05-23T01:47:26.582 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-23T01:47:26.744 [INFO] 
2017-05-23T01:47:26.744 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-core-construction-java ---
2017-05-23T01:47:27.148 [INFO] 
2017-05-23T01:47:27.148 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-core-construction-java ---
2017-05-23T01:47:27.203 [INFO] 
2017-05-23T01:47:27.203 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-core-construction-java ---
2017-05-23T01:47:27.381 [INFO] 
2017-05-23T01:47:27.381 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-core-construction-java ---
2017-05-23T01:47:27.382 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-23T01:47:27.383 [INFO] skip non existing resourceDirectory 

2017-05-23T01:47:27.383 [INFO] Copying 3 resources
2017-05-23T01:47:27.433 [INFO] 
2017-05-23T01:47:27.433 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-core-construction-java ---
2017-05-23T01:47:27.439 [INFO] Changes detected - recompiling the module!
2017-05-23T01:47:27.439 [INFO] Compiling 31 source files to 

2017-05-23T01:47:28.008 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-23T01:47:28.008 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-23T01:47:28.008 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-05-23T01:47:28.008 [INFO] 
:
 Some input files use unchecked or unsafe operations.

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020553#comment-16020553
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3201


> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2142

[Apache Jenkins Server]

See 


Changes:

[klk] Add WindowIntoTranslator

--
[...truncated 404.33 KB...]
2017-05-23T01:47:20.791 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23T01:47:20.791 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23T01:47:20.791 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23T01:47:20.791 [INFO] Including net.bytebuddy:byte-buddy:jar:1.6.8 in 
the shaded jar.
2017-05-23T01:47:20.791 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23T01:47:20.791 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:47:20.791 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:47:20.792 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-05-23T01:47:20.792 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4-M3 from the shaded jar.
2017-05-23T01:47:20.792 [INFO] Including 
org.apache.commons:commons-compress:jar:1.9 in the shaded jar.
2017-05-23T01:47:20.792 [INFO] Including 
org.apache.commons:commons-lang3:jar:3.5 in the shaded jar.
2017-05-23T01:47:20.792 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-05-23T01:47:20.792 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2017-05-23T01:47:20.792 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-05-23T01:47:20.792 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-05-23T01:47:29.186 [INFO] Replacing original artifact with shaded artifact.
2017-05-23T01:47:29.187 [INFO] Replacing 

 with 

2017-05-23T01:47:29.187 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-23T01:47:29.187 [INFO] Replacing 

 with 

2017-05-23T01:47:29.187 [INFO] Dependency-reduced POM written at: 

2017-05-23T01:47:29.378 [INFO] 
2017-05-23T01:47:29.378 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-core ---
2017-05-23T01:47:29.598 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2017-05-23T01:47:30.618 [INFO]  
   
2017-05-23T01:47:30.618 [INFO] 

2017-05-23T01:47:30.618 [INFO] Building Apache Beam :: SDKs :: Java :: IO 
2.1.0-SNAPSHOT
2017-05-23T01:47:30.618 [INFO] 

2017-05-23T01:47:30.621 [INFO] 
2017-05-23T01:47:30.621 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-io-parent ---
2017-05-23T01:47:30.623 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-23T01:47:30.688 [INFO] 
2017-05-23T01:47:30.688 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-sdks-java-io-parent ---
2017-05-23T01:47:30.741 [INFO] 
2017-05-23T01:47:30.741 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-io-parent ---
2017-05-23T01:47:30.795 [INFO] 
2017-05-23T01:47:30.795 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-io-parent ---
2017-05-23T01:47:30.888 [INFO] 
2017-05-23T01:47:30.888 [INFO] --- maven-checkstyle-plugin:2.17:check (default) 
@ beam-sdks-java-io-parent ---
2017-05-23T01:47:31.300 [INFO] Starting audit...
Audit done.
2017-05-23T01:47:31.351 [INFO] 
2017-05-23T01:47:31.352 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-parent ---
2017-05-23T01:47:31.403 [INFO] 
2017-05-23T01:47:31.403 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-parent ---
2017-05-23T01:47:31.504 [INFO] 
2017-05-23T01:47:31.504 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 

[GitHub] beam pull request #3201: [BEAM-1928] Add ReadTranslator

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3201


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add ReadTranslator

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 3c10c0bc8 -> a32bef96d


Add ReadTranslator

This translates Read transforms to ReadPayloads and back


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7f35c98b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7f35c98b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7f35c98b

Branch: refs/heads/master
Commit: 7f35c98b5f77069bd21dc7dea4a7d046883d13a6
Parents: b633abe
Author: Thomas Groh 
Authored: Thu May 18 10:23:35 2017 -0700
Committer: Thomas Groh 
Committed: Mon May 22 16:04:27 2017 -0700

--
 .../core/construction/ReadTranslator.java   | 127 +
 .../core/construction/ReadTranslatorTest.java   | 179 +++
 .../beam/runners/dataflow/ReadTranslator.java   |   5 +-
 .../org/apache/beam/sdk/io/CountingSource.java  |  42 +
 4 files changed, 350 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7f35c98b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslator.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslator.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslator.java
new file mode 100644
index 000..f944938
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslator.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.BytesValue;
+import com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.IsBounded;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.ReadPayload;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.Source;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.SerializableUtils;
+
+/**
+ * Methods for translating {@link Read.Bounded} and {@link Read.Unbounded}
+ * {@link PTransform PTransforms} into {@link ReadPayload} protos.
+ */
+public class ReadTranslator {
+  private static final String JAVA_SERIALIZED_BOUNDED_SOURCE = 
"urn:beam:java:boundedsource:v1";
+  private static final String JAVA_SERIALIZED_UNBOUNDED_SOURCE = 
"urn:beam:java:unboundedsource:v1";
+
+  public static ReadPayload toProto(Read.Bounded read) {
+return ReadPayload.newBuilder()
+.setIsBounded(IsBounded.BOUNDED)
+.setSource(toProto(read.getSource()))
+.build();
+  }
+
+  public static ReadPayload toProto(Read.Unbounded read) {
+return ReadPayload.newBuilder()
+.setIsBounded(IsBounded.UNBOUNDED)
+.setSource(toProto(read.getSource()))
+.build();
+  }
+
+  public static SdkFunctionSpec toProto(Source source) {
+if (source instanceof BoundedSource) {
+  return toProto((BoundedSource) source);
+} else if (source instanceof UnboundedSource) {
+  return toProto((UnboundedSource) source);
+} else {
+  throw new IllegalArgumentException(
+  String.format("Unknown %s type %s", Source.class.getSimpleName(), 
source.getClass()));
+}
+  }
+
+  private static SdkFunctionSpec toProto(BoundedSource source) {
+return SdkFunctionSpec.newBuilder()
+.setSpec(
+FunctionSpec.newBuilder()
+.setUrn(JAVA_SERIALIZED_BOUNDED_SOURCE)
+.setParameter(
+Any.pack(
+

[2/2] beam git commit: This closes #3201: Add ReadTranslator

[kenn]

This closes #3201: Add ReadTranslator


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a32bef96
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a32bef96
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a32bef96

Branch: refs/heads/master
Commit: a32bef96de6dbb4c413176ed5c0b34e4f1a8f221
Parents: 3c10c0b 7f35c98
Author: Kenneth Knowles 
Authored: Mon May 22 18:46:30 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon May 22 18:46:30 2017 -0700

--
 .../core/construction/ReadTranslator.java   | 127 +
 .../core/construction/ReadTranslatorTest.java   | 179 +++
 .../beam/runners/dataflow/ReadTranslator.java   |   5 +-
 .../org/apache/beam/sdk/io/CountingSource.java  |  42 +
 4 files changed, 350 insertions(+), 3 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #21

[Apache Jenkins Server]

See 


Changes:

[tgroh] Include Additional PTransform inputs in Transform Nodes

--
[...truncated 1.06 MB...]
2017-05-23\T\01:42:00.078 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.078 [INFO] Excluding 
io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:42:00.079 [INFO] Excluding 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2901

[Apache Jenkins Server]

See 


Changes:

[klk] Add WindowIntoTranslator

--
[...truncated 400.23 KB...]
2017-05-23T01:44:37.586 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.014 s - in org.apache.beam.sdk.util.SerializableUtilsTest
2017-05-23T01:44:37.588 [INFO] Running 
org.apache.beam.sdk.util.NumberedShardedFileTest
2017-05-23T01:44:37.715 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.125 s - in org.apache.beam.sdk.util.NumberedShardedFileTest
2017-05-23T01:44:37.715 [INFO] Running 
org.apache.beam.sdk.util.UnownedOutputStreamTest
2017-05-23T01:44:37.717 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.UnownedOutputStreamTest
2017-05-23T01:44:37.717 [INFO] Running 
org.apache.beam.sdk.util.CombineFnUtilTest
2017-05-23T01:44:37.731 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.012 s - in org.apache.beam.sdk.util.CombineFnUtilTest
2017-05-23T01:44:37.732 [INFO] Running 
org.apache.beam.sdk.util.BucketingFunctionTest
2017-05-23T01:44:37.734 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.BucketingFunctionTest
2017-05-23T01:44:37.735 [INFO] Running org.apache.beam.sdk.util.NameUtilsTest
2017-05-23T01:44:37.741 [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.004 s - in org.apache.beam.sdk.util.NameUtilsTest
2017-05-23T01:44:37.742 [INFO] Running 
org.apache.beam.sdk.util.UserCodeExceptionTest
2017-05-23T01:44:37.757 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.012 s - in org.apache.beam.sdk.util.UserCodeExceptionTest
2017-05-23T01:44:37.757 [INFO] Running 
org.apache.beam.sdk.util.MovingFunctionTest
2017-05-23T01:44:37.759 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in org.apache.beam.sdk.util.MovingFunctionTest
2017-05-23T01:44:38.331 [INFO] 
2017-05-23T01:44:38.331 [INFO] Results:
2017-05-23T01:44:38.331 [INFO] 
2017-05-23T01:44:38.331 [WARNING] Tests run: 2311, Failures: 0, Errors: 0, 
Skipped: 1
2017-05-23T01:44:38.331 [INFO] 
[JENKINS] Recording test results
2017-05-23T01:45:08.874 [INFO] 
2017-05-23T01:45:08.874 [INFO] --- jacoco-maven-plugin:0.7.8:report (report) @ 
beam-sdks-java-core ---
2017-05-23T01:45:08.983 [INFO] Loading execution data file 

2017-05-23T01:45:09.586 [INFO] Analyzed bundle 'Apache Beam :: SDKs :: Java :: 
Core' with 1037 classes
2017-05-23T01:45:12.095 [INFO] 
2017-05-23T01:45:12.095 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
2017-05-23T01:45:12.235 [INFO] 
2017-05-23T01:45:12.235 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-05-23T01:45:12.455 [INFO] Building jar: 

2017-05-23T01:45:12.722 [INFO] 
2017-05-23T01:45:12.722 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-05-23T01:45:15.286 [INFO] 
2017-05-23T01:45:15.286 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-05-23T01:45:15.452 [INFO] Building jar: 

2017-05-23T01:45:15.615 [INFO] 
2017-05-23T01:45:15.615 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-05-23T01:45:15.623 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2017-05-23T01:45:15.623 [INFO] Including 
com.google.protobuf:protobuf-java:jar:3.2.0 in the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-05-23T01:45:15.623 [INFO] Including net.bytebuddy:byte-buddy:jar:1.6.8 in 
the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-05-23T01:45:15.623 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #21

[Apache Jenkins Server]

See 


Changes:

[tgroh] Include Additional PTransform inputs in Transform Nodes

--
[...truncated 1.06 MB...]
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:36:38.468 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:36:38.469 [INFO] Excluding 

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020544#comment-16020544
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3191


> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3191: [BEAM-1928] Add WindowIntoTranslator

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3191


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add WindowIntoTranslator

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 465f4385b -> 3c10c0bc8


Add WindowIntoTranslator

This translates Window.Assign into a WindowIntoPayload.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bb3db2ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bb3db2ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bb3db2ac

Branch: refs/heads/master
Commit: bb3db2acc07a61844a551ea9f793a0fcf4001878
Parents: 64cea06
Author: Thomas Groh 
Authored: Wed May 17 15:08:50 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon May 22 18:37:26 2017 -0700

--
 .../core/construction/WindowIntoTranslator.java |  61 +
 .../core/construction/WindowingStrategies.java  |  33 ++---
 .../construction/WindowIntoTranslatorTest.java  | 126 +++
 3 files changed, 205 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bb3db2ac/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
new file mode 100644
index 000..ea4c996
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import 
org.apache.beam.runners.core.construction.PTransforms.TransformPayloadTranslator;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+
+/**
+ * Utility methods for translating a {@link Window.Assign} to and from {@link 
RunnerApi}
+ * representations.
+ */
+public class WindowIntoTranslator {
+
+  static class WindowAssignTranslator implements 
TransformPayloadTranslator {
+@Override
+public FunctionSpec translate(
+AppliedPTransform> transform, SdkComponents 
components) {
+  return FunctionSpec.newBuilder()
+  .setUrn("urn:beam:transform:window:v1")
+  .setParameter(
+  Any.pack(WindowIntoTranslator.toProto(transform.getTransform(), 
components)))
+  .build();
+}
+  }
+
+  public static WindowIntoPayload toProto(Window.Assign transform, 
SdkComponents components) {
+return WindowIntoPayload.newBuilder()
+.setWindowFn(WindowingStrategies.toProto(transform.getWindowFn(), 
components))
+.build();
+  }
+
+  public static WindowFn getWindowFn(WindowIntoPayload payload)
+  throws InvalidProtocolBufferException {
+SdkFunctionSpec spec = payload.getWindowFn();
+return WindowingStrategies.windowFnFromProto(spec);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/bb3db2ac/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategies.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategies.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategies.java
index 395702f..8dceebb 100644
--- 

[2/2] beam git commit: This closes #3191: Add WindowIntoTranslator

[kenn]

This closes #3191: Add WindowIntoTranslator


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3c10c0bc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3c10c0bc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3c10c0bc

Branch: refs/heads/master
Commit: 3c10c0bc89783fb6aebb0890ce2fdca2f1b4fb75
Parents: 465f438 bb3db2a
Author: Kenneth Knowles 
Authored: Mon May 22 18:38:27 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon May 22 18:38:27 2017 -0700

--
 .../core/construction/WindowIntoTranslator.java |  61 +
 .../core/construction/WindowingStrategies.java  |  33 ++---
 .../construction/WindowIntoTranslatorTest.java  | 126 +++
 3 files changed, 205 insertions(+), 15 deletions(-)
--

[jira] [Commented] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020543#comment-16020543
 ] 

ASF GitHub Bot commented on BEAM-2345:
--

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/3205

[BEAM-2345] Make versioning in root pom consistent.

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam root-pom-versions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3205


commit c5fe3d06b3d3bba188c75baf3f2862af68a0cb40
Author: Jason Kuster 
Date:   2017-05-23T01:34:28Z

Make versioning in root pom consistent.

Signed-off-by: Jason Kuster 




> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.1.0
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-2345:
--

 Summary: Version configuration of plugins / dependencies in root 
pom.xml is inconsistent
 Key: BEAM-2345
 URL: https://issues.apache.org/jira/browse/BEAM-2345
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Jason Kuster
Assignee: Jason Kuster
Priority: Minor
 Fix For: 2.1.0


Versioning in root pom.xml in some places is controlled by the properties 
section, sometimes is just inline. Move all versioning of plugins / 
dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3205: [BEAM-2345] Make versioning in root pom consistent.

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/3205

[BEAM-2345] Make versioning in root pom consistent.

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam root-pom-versions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3205


commit c5fe3d06b3d3bba188c75baf3f2862af68a0cb40
Author: Jason Kuster 
Date:   2017-05-23T01:34:28Z

Make versioning in root pom consistent.

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #21

[Apache Jenkins Server]

See 


Changes:

[tgroh] Include Additional PTransform inputs in Transform Nodes

--
[...truncated 1.05 MB...]
2017-05-23\T\01:32:09.427 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-05-23\T\01:32:09.427 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-05-23\T\01:32:09.428 [INFO] Excluding 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2141

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2289

[Apache Jenkins Server]

See 


--
[...truncated 575.07 KB...]
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/Extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/Extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s15", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/SideInput-s15.output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s8"
}, 
"user_name": "write/Write/WriteImpl/FinalizeWrite/SideInput-s15"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s16", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/SideInput-s16.output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2900

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1938) Side Inputs should be part of the expanded inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020511#comment-16020511
 ] 

ASF GitHub Bot commented on BEAM-1938:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3188


> Side Inputs should be part of the expanded inputs
> -
>
> Key: BEAM-1938
> URL: https://issues.apache.org/jira/browse/BEAM-1938
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Required for the Java SDK to construct the runner API graphs without 
> inspecting arbitrary transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3188: [BEAM-1938] Include Additional PTransform inputs in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3188


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #3188

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master b633abe2c -> 465f4385b


This closes #3188


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/465f4385
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/465f4385
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/465f4385

Branch: refs/heads/master
Commit: 465f4385b9e0004c468507b1f403c5a2af4bdbb0
Parents: b633abe 9336230
Author: Thomas Groh 
Authored: Mon May 22 18:12:11 2017 -0700
Committer: Thomas Groh 
Committed: Mon May 22 18:12:11 2017 -0700

--
 .../apex/translation/TranslationContext.java|   4 +-
 .../core/construction/TransformInputs.java  |  50 ++
 .../core/construction/TransformInputsTest.java  | 166 +++
 .../beam/runners/direct/DirectGraphVisitor.java |  15 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   9 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 +
 .../beam/runners/direct/WatermarkManager.java   |  17 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../flink/FlinkBatchTranslationContext.java |   3 +-
 .../flink/FlinkStreamingTranslationContext.java |   3 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../beam/sdk/runners/TransformHierarchy.java|  28 +++-
 13 files changed, 280 insertions(+), 28 deletions(-)
--

[2/2] beam git commit: Include Additional PTransform inputs in Transform Nodes

[tgroh]

Include Additional PTransform inputs in Transform Nodes

Add the value of PTransform.getAdditionalInputs in the inputs of a
TransformHierarchy node.

Fork the Node constructor to reduce nullability

This slightly simplifies the constructor implementation(s).

Update the DirectRunner to track main inputs instead of all inputs.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9336230d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9336230d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9336230d

Branch: refs/heads/master
Commit: 9336230d2a5c18bae89908bcd60db8ea96b7906d
Parents: b633abe
Author: Thomas Groh 
Authored: Tue Apr 4 17:43:48 2017 -0700
Committer: Thomas Groh 
Committed: Mon May 22 18:12:11 2017 -0700

--
 .../apex/translation/TranslationContext.java|   4 +-
 .../core/construction/TransformInputs.java  |  50 ++
 .../core/construction/TransformInputsTest.java  | 166 +++
 .../beam/runners/direct/DirectGraphVisitor.java |  15 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   9 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 +
 .../beam/runners/direct/WatermarkManager.java   |  17 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../flink/FlinkBatchTranslationContext.java |   3 +-
 .../flink/FlinkStreamingTranslationContext.java |   3 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../beam/sdk/runners/TransformHierarchy.java|  28 +++-
 13 files changed, 280 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9336230d/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
index aff3863..94d13e1 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
@@ -34,6 +34,7 @@ import 
org.apache.beam.runners.apex.translation.utils.ApexStateInternals;
 import 
org.apache.beam.runners.apex.translation.utils.ApexStateInternals.ApexStateBackend;
 import org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.CoderAdapterStreamCodec;
+import org.apache.beam.runners.core.construction.TransformInputs;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -93,7 +94,8 @@ class TranslationContext {
   }
 
   public  InputT getInput() {
-return (InputT) 
Iterables.getOnlyElement(getCurrentTransform().getInputs().values());
+return (InputT)
+
Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(getCurrentTransform()));
   }
 
   public Map getOutputs() {

http://git-wip-us.apache.org/repos/asf/beam/blob/9336230d/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
new file mode 100644
index 000..2baf93a
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import 

[jira] [Created] (BEAM-2344) Re-rename fat jar in archetypes

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-2344:
--

 Summary: Re-rename fat jar in archetypes
 Key: BEAM-2344
 URL: https://issues.apache.org/jira/browse/BEAM-2344
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Jason Kuster
Assignee: Jason Kuster
 Fix For: 2.1.0


Currently fat jar in archetypes will overwrite the standard jar, and will thus 
be installed as that module's artifact during the install phase. Re-rename it 
such that it is not.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3204: [BEAM-2344] Re-rename fat jar

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/3204

[BEAM-2344] Re-rename fat jar

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam jar-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3204


commit 94cdd31e1e9a386dfcc7964aecf14ffc1c4f5850
Author: Jason Kuster 
Date:   2017-05-23T00:46:49Z

Re-rename fat jar so that install doesn't install the bundled jar as the 
default jar.

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2343) ExecutionContext and StepContext have a lot of unused cruft that obscures their use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020480#comment-16020480
 ] 

ASF GitHub Bot commented on BEAM-2343:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3203

[BEAM-2343] Eliminate ExecutionContext and strip down StepContext to what 
we actually need

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This just deletes a bunch of legacy stuff that _looks_ like shared runner 
support code or APIs but is in fact totally unused or unimplemented.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ExecutionContext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3203.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3203


commit 80321507ffed963b3d37bf91551680b30b1009bd
Author: Kenneth Knowles 
Date:   2017-05-22T22:28:44Z

Move StepContext to top level

commit e357de5fbe548448ca12bc52c6e2530e769620c3
Author: Kenneth Knowles 
Date:   2017-05-22T22:30:33Z

Remove StepContext.noteOutput

commit b2e1a99dff6cd619d53f70f4ee08f173fd7a6777
Author: Kenneth Knowles 
Date:   2017-05-22T22:34:37Z

Rename BaseExecutionContext.StepContext to BaseStepContext

commit 5d91a0d7b6c4831b70276cb5ef67d165b7fd1d05
Author: Kenneth Knowles 
Date:   2017-05-22T22:35:46Z

Move BaseStepContext to the top level

commit 4809fa9bc5c41263836a50e4d281964730ec87c0
Author: Kenneth Knowles 
Date:   2017-05-22T22:36:41Z

Remove extraneous ExecutionContext parameter to BaseStepContext

commit 12f3150395e1c7280d2db43514a09c2a5febfb7b
Author: Kenneth Knowles 
Date:   2017-05-22T22:44:17Z

Implement StepContext directly in the DirectRunner

commit 1b3cb362b4150d227f5f691c779609d5b9bdfbb6
Author: Kenneth Knowles 
Date:   2017-05-22T23:26:00Z

Remove writePCollectionViewData from the Beam codebase

commit e9b5a1c102fcee834c8a3492e15e2e4aa5ae978e
Author: Kenneth Knowles 
Date:   2017-05-22T23:50:41Z

Inline and delete BaseExecutionContext

commit afe57091f72d9b9ad455996454ef04bd5372fde2
Author: Kenneth Knowles 
Date:   2017-05-23T00:33:38Z

Delete unused ExecutionContext




> ExecutionContext and StepContext have a lot of unused cruft that obscures 
> their use
> ---
>
> Key: BEAM-2343
> URL: https://issues.apache.org/jira/browse/BEAM-2343
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Like {{noteOutput}} and pretty much the existence of {{ExecutionContext}} at 
> all.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3203: [BEAM-2343] Eliminate ExecutionContext and strip do...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3203

[BEAM-2343] Eliminate ExecutionContext and strip down StepContext to what 
we actually need

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This just deletes a bunch of legacy stuff that _looks_ like shared runner 
support code or APIs but is in fact totally unused or unimplemented.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ExecutionContext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3203.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3203


commit 80321507ffed963b3d37bf91551680b30b1009bd
Author: Kenneth Knowles 
Date:   2017-05-22T22:28:44Z

Move StepContext to top level

commit e357de5fbe548448ca12bc52c6e2530e769620c3
Author: Kenneth Knowles 
Date:   2017-05-22T22:30:33Z

Remove StepContext.noteOutput

commit b2e1a99dff6cd619d53f70f4ee08f173fd7a6777
Author: Kenneth Knowles 
Date:   2017-05-22T22:34:37Z

Rename BaseExecutionContext.StepContext to BaseStepContext

commit 5d91a0d7b6c4831b70276cb5ef67d165b7fd1d05
Author: Kenneth Knowles 
Date:   2017-05-22T22:35:46Z

Move BaseStepContext to the top level

commit 4809fa9bc5c41263836a50e4d281964730ec87c0
Author: Kenneth Knowles 
Date:   2017-05-22T22:36:41Z

Remove extraneous ExecutionContext parameter to BaseStepContext

commit 12f3150395e1c7280d2db43514a09c2a5febfb7b
Author: Kenneth Knowles 
Date:   2017-05-22T22:44:17Z

Implement StepContext directly in the DirectRunner

commit 1b3cb362b4150d227f5f691c779609d5b9bdfbb6
Author: Kenneth Knowles 
Date:   2017-05-22T23:26:00Z

Remove writePCollectionViewData from the Beam codebase

commit e9b5a1c102fcee834c8a3492e15e2e4aa5ae978e
Author: Kenneth Knowles 
Date:   2017-05-22T23:50:41Z

Inline and delete BaseExecutionContext

commit afe57091f72d9b9ad455996454ef04bd5372fde2
Author: Kenneth Knowles 
Date:   2017-05-23T00:33:38Z

Delete unused ExecutionContext




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2343) ExecutionContext and StepContext have a lot of unused cruft that obscures their use

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2343:
-

 Summary: ExecutionContext and StepContext have a lot of unused 
cruft that obscures their use
 Key: BEAM-2343
 URL: https://issues.apache.org/jira/browse/BEAM-2343
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Like {{noteOutput}} and pretty much the existence of {{ExecutionContext}} at 
all.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #20

[Apache Jenkins Server]

See 


--
[...truncated 1.42 MB...]
2017-05-23\T\00:33:31.220 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from 
the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 
from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-common:jar:2.1.0-SNAPSHOT from the 
shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.1.0-SNAPSHOT from 
the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hbase:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.hbase:hbase-shaded-client:jar:1.2.5 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.htrace:htrace-core:jar:3.1.0-incubating from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1 from the shaded 
jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-jdbc:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.commons:commons-dbcp2:jar:2.1.1 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.commons:commons-pool2:jar:2.4.2 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-jms:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1 from the shaded jar.
2017-05-23\T\00:33:31.220 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-kafka:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.apache.kafka:kafka-clients:jar:0.9.0.1 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.springframework:spring-expression:jar:4.3.5.RELEASE from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.springframework:spring-core:jar:4.3.5.RELEASE from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-kinesis:jar:2.1.0-SNAPSHOT from the shaded 
jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-kinesis:jar:1.11.18 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:amazon-kinesis-client:jar:1.6.1 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-dynamodb:jar:1.10.20 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-s3:jar:1.10.20 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-kms:jar:1.10.20 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-cloudwatch:jar:1.10.20 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding commons-lang:commons-lang:jar:2.6 
from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.amazonaws:aws-java-sdk-core:jar:1.11.18 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.6 from the 
shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-mongodb:jar:2.1.0-SNAPSHOT from the shaded 
jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.mongodb:mongo-java-driver:jar:3.2.2 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.apache.beam:beam-sdks-java-io-mqtt:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] Excluding 
org.fusesource.mqtt-client:mqtt-client:jar:1.14 from the shaded jar.
2017-05-23\T\00:33:31.221 [INFO] 

[jira] [Commented] (BEAM-2300) Upgrade python sdk dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020412#comment-16020412
 ] 

ASF GitHub Bot commented on BEAM-2300:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3202

[BEAM-2300] upgrading python sdk dependencies

R: @charlesccychen 

For pinned dependencies, I switched them to range dependencies as otherwise 
we would break backward compatibility.

For range dependencies, I extended the range.

It will be trick to test all these ranges with all combinations of 
dependencies and ensure that things are in working order.

cc: @sb2nov @markflyhigh for ideas.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam verd

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3202


commit ade0e1a119cc83af0363f85f1ade4a94d1ff5597
Author: Ahmet Altay 
Date:   2017-05-22T23:40:09Z

upgrading python sdk dependencies




> Upgrade python sdk dependencies
> ---
>
> Key: BEAM-2300
> URL: https://issues.apache.org/jira/browse/BEAM-2300
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
> Fix For: 2.1.0
>
>
> cc: [~charleschen]
> Upgraded python sdk dependencies if possible:
> Some opportunities:
> protobuf 3.3.0
> proto-google-cloud-datastore-v1 0.90.4
> google-apitools 0.5.11
> google-cloud-bigquery 0.24.0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3202: [BEAM-2300] upgrading python sdk dependencies

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3202

[BEAM-2300] upgrading python sdk dependencies

R: @charlesccychen 

For pinned dependencies, I switched them to range dependencies as otherwise 
we would break backward compatibility.

For range dependencies, I extended the range.

It will be trick to test all these ranges with all combinations of 
dependencies and ensure that things are in working order.

cc: @sb2nov @markflyhigh for ideas.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam verd

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3202


commit ade0e1a119cc83af0363f85f1ade4a94d1ff5597
Author: Ahmet Altay 
Date:   2017-05-22T23:40:09Z

upgrading python sdk dependencies




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1158) Create HiveIO

[Seshadri Raghunathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020404#comment-16020404
 ] 

Seshadri Raghunathan commented on BEAM-1158:


Reading from Hive works with HadoopInputFormatIO. Please refer to the SO post - 
https://stackoverflow.com/questions/44123565/how-do-i-read-from-hive-using-apache-beam

I plan to do a write-up for the same here - 
https://beam.apache.org/documentation/io/built-in/hadoop/

Given that the read works as described, does it make sense to close this 
ticket. Please let me know if there are any concerns !

Thanks,
Seshadri

PS:
I am also working on a native-API source & sink implementation for Hive which 
can be tagged against a new JIRA.



> Create HiveIO
> -
>
> Key: BEAM-1158
> URL: https://issues.apache.org/jira/browse/BEAM-1158
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
>
> Support for reading and writing to Hive. The HiveIO will directly use the 
> native API and HiveQL



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3188

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020366#comment-16020366
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3201

[BEAM-1928] Add ReadTranslator


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This translates Read transforms to ReadPayloads and back


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam read_translator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3201


commit 7f35c98b5f77069bd21dc7dea4a7d046883d13a6
Author: Thomas Groh 
Date:   2017-05-18T17:23:35Z

Add ReadTranslator

This translates Read transforms to ReadPayloads and back




> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3201: [BEAM-1928] Add ReadTranslator

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3201

[BEAM-1928] Add ReadTranslator


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This translates Read transforms to ReadPayloads and back


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam read_translator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3201


commit 7f35c98b5f77069bd21dc7dea4a7d046883d13a6
Author: Thomas Groh 
Date:   2017-05-18T17:23:35Z

Add ReadTranslator

This translates Read transforms to ReadPayloads and back




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2288

[Apache Jenkins Server]

See 


Changes:

[klk] Remove Pipeline reference from TransformHierarchy

--
[...truncated 802.25 KB...]
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:even/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s30"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s32", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": 

[GitHub] beam pull request #3200: Adding a snippet for metrics

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/3200

Adding a snippet for metrics

Adding a snippet to use in metrics documentation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam snippet-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3200


commit 46c1137948e9684accba333ae7921992c59c4659
Author: Pablo 
Date:   2017-05-22T22:39:44Z

Adding a snippet for metrics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020329#comment-16020329
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3199

[BEAM-1928] Add CombineTranslator

This translates Combines to CombinePayloads and back

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam combine_translator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3199


commit 38dd04c135cf6bf00ff31015e1c66f1edbd4a6ea
Author: Thomas Groh 
Date:   2017-05-18T17:23:16Z

Add CombineTranslator

This translates Combines to CombinePayloads and back




> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3199: [BEAM-1928] Add CombineTranslator

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3199

[BEAM-1928] Add CombineTranslator

This translates Combines to CombinePayloads and back

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam combine_translator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3199


commit 38dd04c135cf6bf00ff31015e1c66f1edbd4a6ea
Author: Thomas Groh 
Date:   2017-05-18T17:23:16Z

Add CombineTranslator

This translates Combines to CombinePayloads and back




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020290#comment-16020290
 ] 

ASF GitHub Bot commented on BEAM-2333:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3187


> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #3187: Remove Pipeline reference from TransformHierarchy

[kenn]

This closes #3187: Remove Pipeline reference from TransformHierarchy


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b633abe2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b633abe2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b633abe2

Branch: refs/heads/master
Commit: b633abe2c8efd3ac607a0be14980315e5f0d9fa6
Parents: 983a449 5e9fceb
Author: Kenneth Knowles 
Authored: Mon May 22 15:16:22 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon May 22 15:16:22 2017 -0700

--
 .../translation/ApexPipelineTranslator.java |  4 +-
 .../apex/translation/TranslationContext.java|  5 +-
 .../core/construction/SdkComponents.java| 14 +++---
 .../core/construction/SdkComponentsTest.java|  7 +--
 .../beam/runners/direct/DirectGraphVisitor.java |  3 +-
 .../direct/KeyedPValueTrackingVisitor.java  |  5 +-
 .../flink/FlinkBatchPipelineTranslator.java |  2 +-
 .../apache/beam/runners/flink/FlinkRunner.java  | 10 +---
 .../flink/FlinkStreamingPipelineTranslator.java |  4 +-
 .../dataflow/DataflowPipelineTranslator.java|  6 +--
 .../beam/runners/dataflow/DataflowRunner.java   |  2 +-
 .../apache/beam/runners/spark/SparkRunner.java  |  2 +-
 .../streaming/TrackStreamingSourcesTest.java| 14 +-
 .../main/java/org/apache/beam/sdk/Pipeline.java | 50 
 .../beam/sdk/runners/TransformHierarchy.java|  6 +--
 .../sdk/runners/TransformHierarchyTest.java |  2 +-
 16 files changed, 82 insertions(+), 54 deletions(-)
--

[GitHub] beam pull request #3187: [BEAM-2333] Remove Pipeline reference from Transfor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3187


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove Pipeline reference from TransformHierarchy

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 983a44926 -> b633abe2c


Remove Pipeline reference from TransformHierarchy

This change removes a direct dependency cycle between Pipeline and
TransformHierarchy. There is still an indirect cycle through PValues, but that
is slightly less problematic.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5e9fcebc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5e9fcebc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5e9fcebc

Branch: refs/heads/master
Commit: 5e9fcebc07725de368391914781e5b4d5f9c4a19
Parents: d7a4e49
Author: Kenneth Knowles 
Authored: Fri May 19 12:57:41 2017 -0700
Committer: Kenneth Knowles 
Committed: Sat May 20 08:33:21 2017 -0700

--
 .../translation/ApexPipelineTranslator.java |  4 +-
 .../apex/translation/TranslationContext.java|  5 +-
 .../core/construction/SdkComponents.java| 14 +++---
 .../core/construction/SdkComponentsTest.java|  7 +--
 .../beam/runners/direct/DirectGraphVisitor.java |  3 +-
 .../direct/KeyedPValueTrackingVisitor.java  |  5 +-
 .../flink/FlinkBatchPipelineTranslator.java |  2 +-
 .../apache/beam/runners/flink/FlinkRunner.java  | 10 +---
 .../flink/FlinkStreamingPipelineTranslator.java |  4 +-
 .../dataflow/DataflowPipelineTranslator.java|  6 +--
 .../beam/runners/dataflow/DataflowRunner.java   |  2 +-
 .../apache/beam/runners/spark/SparkRunner.java  |  2 +-
 .../streaming/TrackStreamingSourcesTest.java| 14 +-
 .../main/java/org/apache/beam/sdk/Pipeline.java | 50 
 .../beam/sdk/runners/TransformHierarchy.java|  6 +--
 .../sdk/runners/TransformHierarchyTest.java |  2 +-
 16 files changed, 82 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5e9fcebc/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ApexPipelineTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ApexPipelineTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ApexPipelineTranslator.java
index 32e470f..bda074b 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ApexPipelineTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ApexPipelineTranslator.java
@@ -49,7 +49,7 @@ import org.slf4j.LoggerFactory;
  * into Apex logical plan {@link DAG}.
  */
 @SuppressWarnings({"rawtypes", "unchecked"})
-public class ApexPipelineTranslator implements Pipeline.PipelineVisitor {
+public class ApexPipelineTranslator extends Pipeline.PipelineVisitor.Defaults {
   private static final Logger LOG = 
LoggerFactory.getLogger(ApexPipelineTranslator.class);
 
   /**
@@ -110,7 +110,7 @@ public class ApexPipelineTranslator implements 
Pipeline.PipelineVisitor {
   throw new UnsupportedOperationException(
   "no translator registered for " + transform);
 }
-translationContext.setCurrentTransform(node);
+
translationContext.setCurrentTransform(node.toAppliedPTransform(getPipeline()));
 translator.translate(transform, translationContext);
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/5e9fcebc/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
index a5e3028..aff3863 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
@@ -36,7 +36,6 @@ import 
org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.CoderAdapterStreamCodec;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.windowing.Window;
@@ -77,8 +76,8 @@ class TranslationContext {
 this.pipelineOptions = pipelineOptions;
   }
 
-  public void setCurrentTransform(TransformHierarchy.Node treeNode) {
-this.currentTransform = treeNode.toAppliedPTransform();
+  public void setCurrentTransform(AppliedPTransform transform) {
+this.currentTransform = transform;
   }
 
   public ApexPipelineOptions getPipelineOptions() {

Build failed in Jenkins: beam_PostCommit_Python_Verify #2287

[Apache Jenkins Server]

See 


Changes:

[altay] [BEAM-1999] Remove resolved todo from value providers

[altay] Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6

--
[...truncated 611.48 KB...]
  File "apache_beam/runners/worker/operations.py", line 338, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11076)
with self.scoped_process_state:
  File "apache_beam/runners/worker/operations.py", line 339, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11030)
self.dofn_receiver.receive(o)
  File "apache_beam/runners/common.py", line 382, in 
apache_beam.runners.common.DoFnRunner.receive 
(apache_beam/runners/common.c:10156)
self.process(windowed_value)
  File "apache_beam/runners/common.py", line 390, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10458)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 415, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:11363)
raise
  File "apache_beam/runners/common.py", line 388, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10371)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 189, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:6270)
self.output_processor.process_outputs(
  File "apache_beam/runners/common.py", line 480, in 
apache_beam.runners.common._OutputProcessor.process_outputs 
(apache_beam/runners/common.c:12500)
self.main_receivers.receive(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 86, in 
apache_beam.runners.worker.operations.ConsumerSet.receive 
(apache_beam/runners/worker/operations.c:3622)
cython.cast(Operation, consumer).process(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 338, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11076)
with self.scoped_process_state:
  File "apache_beam/runners/worker/operations.py", line 339, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11030)
self.dofn_receiver.receive(o)
  File "apache_beam/runners/common.py", line 382, in 
apache_beam.runners.common.DoFnRunner.receive 
(apache_beam/runners/common.c:10156)
self.process(windowed_value)
  File "apache_beam/runners/common.py", line 390, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10458)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 415, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:11363)
raise
  File "apache_beam/runners/common.py", line 388, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10371)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 189, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:6270)
self.output_processor.process_outputs(
  File "apache_beam/runners/common.py", line 480, in 
apache_beam.runners.common._OutputProcessor.process_outputs 
(apache_beam/runners/common.c:12500)
self.main_receivers.receive(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 86, in 
apache_beam.runners.worker.operations.ConsumerSet.receive 
(apache_beam/runners/worker/operations.c:3622)
cython.cast(Operation, consumer).process(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 338, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11076)
with self.scoped_process_state:
  File "apache_beam/runners/worker/operations.py", line 339, in 
apache_beam.runners.worker.operations.DoOperation.process 
(apache_beam/runners/worker/operations.c:11030)
self.dofn_receiver.receive(o)
  File "apache_beam/runners/common.py", line 382, in 
apache_beam.runners.common.DoFnRunner.receive 
(apache_beam/runners/common.c:10156)
self.process(windowed_value)
  File "apache_beam/runners/common.py", line 390, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10458)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 431, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:11673)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 388, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:10371)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 281, in 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3880

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3187

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #18

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2286

[Apache Jenkins Server]

See 


Changes:

[tgroh] Cleanup k8s scripts naming & don't create insecure svc by default

--
[...truncated 814.20 KB...]
"user_name": "assert_that/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"display_data": [
  {
"key": "fn", 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2897

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2896

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #17

[Apache Jenkins Server]

See 


Changes:

[mingmxu] update JavaDoc for BoundedWindow

[vikasrk] Triggers for Java SDK post submit tests

[robertwb] Deprecate .options usage

[lcwik] Remove Coder.asCloudObject and related methods

[altay] Updating Dataflow API protos and client

[tgroh] Add Additional CloudObjectTranslators

[kirpichov] Removed coder and parseFn from PubsubIO.Read

[kirpichov] Removed coder and formatFn from PubsubIO.Write

[kirpichov] Introduces read/writePubsubMessages

[klk] Use LinkedHashMap for step contexts in BaseExecutionContext

[dhalperi] Remove IoChannelUtils from PackageUtil

[klk] Use Thread context ClassLoader as parent in AvroCoderTest

[dhalperi] [BEAM-59] FileChecksumMatcherTest: switch from IOChannelUtils to java

[tgroh] Mark all PCollectionView methods internal

[lcwik] [BEAM-1871] Move GCP specific serialization CloudObject and supporting

[tgroh] Reduce visibility of DirectRunner classes

[altay] Retry on correct error codes for datastoreio

[robertwb] Try PTransform-based coder inference before using fallback coder.

[robertwb] Fix coder inference ambiguity in CombineFnsTest.

[klk] Remove job name usages from BigQueryIO at pipeline construction time

[kirpichov] Fully general dynamic tables (including schemas) in BigQueryIO.

[kirpichov] Fixes PubsubIO.Write translation in Dataflow runner

[dhalperi] Convert WriteFiles/FileBasedSink from IOChannelFactory to FileSystems

[tgroh] Inline TypedPValue

[lcwik] [BEAM-2157] Fix HDFS Test That Failed in Other JDK Versions

[kirpichov] Provides a default coder for PubsubMessage

[kirpichov] Moves PubsubMessage to upper level and renames payload

[klk] BigQueryIO: Remove tempLocation usage at pipeline construction time

[dhalperi] Removal of Aggregator class. Also removal from comments.

[klk] Annotate internal methods of PCollection

[klk] Annotate internal pieces of sdks.transforms

[klk] Tighten access control and internal annotations for triggers

[klk] The transforms.reflect package is not for users

[klk] Annotate internal-only bits of Java sdk.runners

[klk] Tighten access in sdk.options

[klk] Annotate internal methods on Pipeline

[lcwik] [BEAM-2165] Update Apex to support serializing/deserializing custom user

[lcwik] [BEAM-2165] Update Dataflow to support serializing/deserializing custom

[lcwik] [BEAM-2165] Update Flink to support serializing/deserializing custom

[lcwik] [BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

[dhalperi] DataflowRunner: integration test GCP-IO

[lcwik] [BEAM-1491]Identify HADOOP_CONF_DIR(or YARN_CONF_DIR) environment

[lcwik] Split Coder's encode/decode methods into two methods depending on

[dhalperi] [BEAM-2162] Add logging to long BigQuery jobs

[tgroh] Use BinaryCombineLongFn in GroupIntoBatches

[altay] [BEAM-2152] Remove gcloud auth as application default credentials does

[dhalperi] AvroIOTest: stop using IOChannelUtils, remove invalid test

[dhalperi] DataflowRunner: switch from IOChannels to FileSystems for creating 
files

[dhalperi] DataflowPipelineOptions: use FileSystems, not IOChannelUtils, to 
resolve

[dhalperi] Stop registering IOChannelFactories in SDK harness, runners, tests

[dhalperi] FileSystems: add match(String) for single spec

[dhalperi] WriteWithShardingFactoryTest: switch to FileSystems

[dhalperi] ReadTranslator: remove outdated comment

[dhalperi] TFRecordIO: switch to FileSystems from IOChannelUtils

[lcwik] Remove POutput#recordAsOutput

[robertwb] One more TaggedOutput name change.

[dhalperi] Update JavaDoc of checkpoint, and expose `enable.auto.commit` for 
more

[lcwik] Use CloudObject encodings in the Beam Fn Harness

[aviemzur] [BEAM-1763] Verify PAssert execution in runners which support 
metrics.

[aviemzur] [BEAM-1726] Fix empty side inputs in Flink Streaming Runner

[aviemzur] [BEAM-1726] Fix RuntimeException throwing in FlinkStateInternals

[aviemzur] [BEAM-1726] Fix Flatten with input copies in Flink Streaming Runner

[aviemzur] [BEAM-1763] Replace usage of PipelineRule with TestPipeline in Spark

[dhalperi] GCP-IO: manage versions in top level pom

[dhalperi] Remove IOChannelUtil/Factory from BigQueryIO

[klk] Eliminate Pipeline.getOptions

[robertwb] Cleanup old cython files before testing again

[klk] [BEAM-1727] Add align and offset to Timer

[klk] Fix initial watermark of DoFnOperator in Flink runner

[dhalperi] [BEAM-59] Remove IOChannelUtils, IOChannelFactory

[dhalperi] Move thrown.expect to correct place

[tgroh] Note when checkpoints may be committed.

[klk] Activate WindowedWordCountIT on Apex runner

[tgroh] Split StartBundleContext and FinishBundleContext

[kirpichov] [BEAM-2170] PubsubMessageWithAttributesCoder should not NPE on 
messages

[kirpichov] Makes PubsubMessagePayloadOnlyCoder not require whole-stream context

[kirpichov] Moves coder choice into PubsubSource

[klk] Move Java 

[1/2] beam git commit: Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 509baac58 -> 983a44926


Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/39e8d37e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/39e8d37e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/39e8d37e

Branch: refs/heads/master
Commit: 39e8d37ee95709d07a74c9df866059871d5585f5
Parents: 509baac
Author: Charles Chen 
Authored: Mon May 22 13:03:38 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 22 13:55:39 2017 -0700

--
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/39e8d37e/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 0ecd22a..a05e582 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -62,7 +62,7 @@ class DataflowRunner(PipelineRunner):
   # Environment version information. It is passed to the service during a
   # a job submission and is used by the service to establish what features
   # are expected by the workers.
-  BATCH_ENVIRONMENT_MAJOR_VERSION = '5'
+  BATCH_ENVIRONMENT_MAJOR_VERSION = '6'
   STREAMING_ENVIRONMENT_MAJOR_VERSION = '0'
 
   def __init__(self, cache=None):

[GitHub] beam pull request #3198: Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3198


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1999) Change signature of set_runtime_options in value_provider.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020186#comment-16020186
 ] 

ASF GitHub Bot commented on BEAM-1999:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3197


> Change signature of set_runtime_options in value_provider.py
> 
>
> Key: BEAM-1999
> URL: https://issues.apache.org/jira/browse/BEAM-1999
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: María GH
> Fix For: Not applicable
>
>
> Change signature of this function to remove _unused_options_id, and add 
> kwargs to make it flexible for future changes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #3198

[altay]

This closes #3198


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/983a4492
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/983a4492
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/983a4492

Branch: refs/heads/master
Commit: 983a449268559a09962f9c2a4c60953b008d8052
Parents: 509baac 39e8d37
Author: Ahmet Altay 
Authored: Mon May 22 13:55:44 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 22 13:55:44 2017 -0700

--
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #3197: [BEAM-1999] Remove resolved todo from value provide...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3197


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3197

[altay]

This closes #3197


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/509baac5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/509baac5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/509baac5

Branch: refs/heads/master
Commit: 509baac58a6972ee0f5101f14d84190b768b138a
Parents: fff82b0 13d2012
Author: Ahmet Altay 
Authored: Mon May 22 13:55:06 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 22 13:55:06 2017 -0700

--
 sdks/python/apache_beam/options/value_provider.py | 1 -
 1 file changed, 1 deletion(-)
--

[1/2] beam git commit: [BEAM-1999] Remove resolved todo from value providers

[altay]

Repository: beam
Updated Branches:
  refs/heads/master fff82b003 -> 509baac58


[BEAM-1999] Remove resolved todo from value providers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/13d20122
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/13d20122
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/13d20122

Branch: refs/heads/master
Commit: 13d20122bc0b20e1348714409fbc3b6b1f1c1e4f
Parents: fff82b0
Author: Sourabh Bajaj 
Authored: Mon May 22 13:00:52 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 22 13:54:59 2017 -0700

--
 sdks/python/apache_beam/options/value_provider.py | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/13d20122/sdks/python/apache_beam/options/value_provider.py
--
diff --git a/sdks/python/apache_beam/options/value_provider.py 
b/sdks/python/apache_beam/options/value_provider.py
index 40bddba..f7630bd 100644
--- a/sdks/python/apache_beam/options/value_provider.py
+++ b/sdks/python/apache_beam/options/value_provider.py
@@ -82,7 +82,6 @@ class RuntimeValueProvider(ValueProvider):
   value = self.default_value
 return value
 
-  # TODO(BEAM-1999): Remove _unused_options_id
   @classmethod
   def set_runtime_options(cls, pipeline_options):
 RuntimeValueProvider.runtime_options = pipeline_options

[jira] [Commented] (BEAM-1909) BigQuery read transform fails for DirectRunner when querying non-US regions

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020138#comment-16020138
 ] 

Chamikara Jayalath commented on BEAM-1909:
--

Hi Uwe,

The issue here is not creating a table/dataset in the correct region for 
executing queries for non-US regions. We can do this if we know the region.
 
You can determine the region using a dry-run of the query. When you dry-run the 
query you should get statistics about the query including the set of tables 
referenced. This information can be used to determine the region (pick one of 
the tables and get the dataset).

An example response to a dry-run query is given below.


 tableId: u'temp_table_a33db6e872704a238568ba77f6b55e80'>
 flattenResults: True
 query: u'SELECT * FROM [clouddataflow-readonly:samples.weather_stations];'
 useLegacySql: True
 userDefinedFunctionResources: []>>
 etag: u'"a19Ag3pFAfODmeUgLyZLAGQkEbw/9Uc9pvfbc0cqTPPIgjuvzASt7UY"'
 jobReference: >
 kind: u'bigquery#job'
 statistics: ]
 totalBytesBilled: 0
 totalBytesProcessed: 151184>
 totalBytesProcessed: 151184>
 status: 
 user_email: '>

- Cham

> BigQuery read transform fails for DirectRunner when querying non-US regions
> ---
>
> Key: BEAM-1909
> URL: https://issues.apache.org/jira/browse/BEAM-1909
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>
> See: 
> http://stackoverflow.com/questions/42135002/google-dataflow-cannot-read-and-write-in-different-locations-python-sdk-v0-5-5/42144748?noredirect=1#comment73621983_42144748
> This should be fixed by creating the temp dataset and table in the correct 
> region.
> cc: [~sb2nov]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1724) ApexRunner should deduplicate when an UnboundedSource requires Deduping

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020107#comment-16020107
 ] 

Kenneth Knowles commented on BEAM-1724:
---

I removed the release version just to avoid having all issues float along from 
release to release without explicit decision to do so. If it helps your own 
tracking, please feel free to re-add a future release.

> ApexRunner should deduplicate when an UnboundedSource requires Deduping
> ---
>
> Key: BEAM-1724
> URL: https://issues.apache.org/jira/browse/BEAM-1724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Groh
>
> The Apex implementation of UnboundedRead does not inspect the 
> requiresDeduping property of an Unbounded Source. It should do so.
> https://github.com/apache/beam/blob/master/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2342) k8s scripts - must not create publicly visible service by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020105#comment-16020105
 ] 

ASF GitHub Bot commented on BEAM-2342:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3196


> k8s scripts - must not create publicly visible service by default
> -
>
> Key: BEAM-2342
> URL: https://issues.apache.org/jira/browse/BEAM-2342
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> The k8s scripts for cassandra and elasticsearch setup a internet-accessible 
> service by default in start-up.sh, which is insecure since we rely on 
> firewalls for securing the data stores. 
> We *should* keep these kubernetes scripts for doing local development work 
> (-local-dev.yaml), but we shouldn't set them up by default in the script.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3196: [BEAM-2342] Cleanup k8s scripts naming & don't crea...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3196


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #3196

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 996782b62 -> fff82b003


This closes #3196


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fff82b00
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fff82b00
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fff82b00

Branch: refs/heads/master
Commit: fff82b003879fc4b914b88ddff0582b50b3513a5
Parents: 996782b 43e2910
Author: Thomas Groh 
Authored: Mon May 22 13:11:03 2017 -0700
Committer: Thomas Groh 
Committed: Mon May 22 13:11:03 2017 -0700

--
 .../cassandra/LargeITCluster/setup.sh   | 21 ++
 .../cassandra/LargeITCluster/start-up.sh| 22 ---
 .../cassandra/LargeITCluster/teardown.sh|  1 -
 .../cassandra/SmallITCluster/setup.sh   | 22 +++
 .../cassandra/SmallITCluster/start-up.sh| 23 
 .../cassandra/SmallITCluster/teardown.sh|  1 -
 .../LargeProductionCluster/setup.sh | 21 ++
 .../LargeProductionCluster/start-up.sh  | 22 ---
 .../LargeProductionCluster/teardown.sh  |  1 -
 .../elasticsearch/SmallITCluster/setup.sh   | 22 +++
 .../elasticsearch/SmallITCluster/start-up.sh| 23 
 .../elasticsearch/SmallITCluster/teardown.sh|  1 -
 12 files changed, 86 insertions(+), 94 deletions(-)
--

[2/2] beam git commit: Cleanup k8s scripts naming & don't create insecure svc by default

[tgroh]

Cleanup k8s scripts naming & don't create insecure svc by default

These scripts setup a internet-accessible service by default, which is
insecure since we rely on firewalls for securing the data stores.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/43e29101
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/43e29101
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/43e29101

Branch: refs/heads/master
Commit: 43e291015b07fe8f98cb70131bb2482dfa633c7c
Parents: 996782b
Author: Stephen Sisk 
Authored: Mon May 22 11:36:09 2017 -0700
Committer: Thomas Groh 
Committed: Mon May 22 13:11:03 2017 -0700

--
 .../cassandra/LargeITCluster/setup.sh   | 21 ++
 .../cassandra/LargeITCluster/start-up.sh| 22 ---
 .../cassandra/LargeITCluster/teardown.sh|  1 -
 .../cassandra/SmallITCluster/setup.sh   | 22 +++
 .../cassandra/SmallITCluster/start-up.sh| 23 
 .../cassandra/SmallITCluster/teardown.sh|  1 -
 .../LargeProductionCluster/setup.sh | 21 ++
 .../LargeProductionCluster/start-up.sh  | 22 ---
 .../LargeProductionCluster/teardown.sh  |  1 -
 .../elasticsearch/SmallITCluster/setup.sh   | 22 +++
 .../elasticsearch/SmallITCluster/start-up.sh| 23 
 .../elasticsearch/SmallITCluster/teardown.sh|  1 -
 12 files changed, 86 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/43e29101/.test-infra/kubernetes/cassandra/LargeITCluster/setup.sh
--
diff --git a/.test-infra/kubernetes/cassandra/LargeITCluster/setup.sh 
b/.test-infra/kubernetes/cassandra/LargeITCluster/setup.sh
new file mode 100644
index 000..7bc0809
--- /dev/null
+++ b/.test-infra/kubernetes/cassandra/LargeITCluster/setup.sh
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/bin/bash
+set -e
+
+# Create Cassandra services and statefulset.
+kubectl create -f cassandra-svc-statefulset.yaml

http://git-wip-us.apache.org/repos/asf/beam/blob/43e29101/.test-infra/kubernetes/cassandra/LargeITCluster/start-up.sh
--
diff --git a/.test-infra/kubernetes/cassandra/LargeITCluster/start-up.sh 
b/.test-infra/kubernetes/cassandra/LargeITCluster/start-up.sh
deleted file mode 100644
index 7341209..000
--- a/.test-infra/kubernetes/cassandra/LargeITCluster/start-up.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-#!/bin/bash
-set -e
-
-# Create Cassandra services and statefulset.
-kubectl create -f cassandra-service-for-local-dev.yaml
-kubectl create -f cassandra-svc-statefulset.yaml

http://git-wip-us.apache.org/repos/asf/beam/blob/43e29101/.test-infra/kubernetes/cassandra/LargeITCluster/teardown.sh
--
diff --git a/.test-infra/kubernetes/cassandra/LargeITCluster/teardown.sh 
b/.test-infra/kubernetes/cassandra/LargeITCluster/teardown.sh
index 367b604..3d040a6 100644
--- a/.test-infra/kubernetes/cassandra/LargeITCluster/teardown.sh
+++ 

[jira] [Comment Edited] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014562#comment-16014562
 ] 

Kenneth Knowles edited comment on BEAM-981 at 5/22/17 8:09 PM:
---

I think we should drop the fix version for work that is ongoing, and set it up 
when it gets fixed. The first stable release needed an explicit burndown, but 
going forward releases should be mostly time based.

If the version helps your own tracking, or you want to be sure the release 
manager for 2.1.0 tracks it, please feel free to re-add a fix version.


was (Author: kenn):
I think we should drop the fix version for work that is ongoing, and set it up 
when it gets fixed. The first stable release needed an explicit burndown, but 
going forward releases should be mostly time based.

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Kobi Salant
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> 

[jira] [Commented] (BEAM-2321) gRPC configuration failure using DataflowRunner and Bigtable

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020097#comment-16020097
 ] 

Daniel Halperin commented on BEAM-2321:
---

By the way, the reason I believe this is not a bug in the SDK is that we have 
running integration tests that use Bigtable on Dataflow service which are 
passing.

> gRPC configuration failure using DataflowRunner and Bigtable
> 
>
> Key: BEAM-2321
> URL: https://issues.apache.org/jira/browse/BEAM-2321
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0
>Reporter: Nigel Kilmer
>Assignee: Daniel Halperin
>
> I'm attempting to run a pipeline that uses the DataflowRunner and writes to 
> Bigtable (v0.9.6.2). This exception is thrown (looks like it's when the 
> BigtableSession is being created):
> java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly 
> configured.
>   at 
> io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139)
>   at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.createSslContext(BigtableSession.java:124)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.access$000(BigtableSession.java:81)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession$2.run(BigtableSession.java:151)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I can run the same pipeline without issue using the DirectRunner instead, and 
> it was also working for me using the 0.7.0 snapshot of Beam last week. I've 
> already checked with the cloud-bigtable-client project; they said that it 
> should be working since I have a dependency on netty_tcnative configured. The 
> fact that the same pipeline works with the DirectRunner and not with the 
> DataflowRunner makes me think it's a DataflowRunner bug.
> My pipeline is pretty simple; it looks like this:
> Pipeline p = Pipeline.create(gcpOptions);
> p.apply(PubsubIO.readProtos(TestProto.class)
> .fromSubscription(pubsubSubscription))
>   .apply(ParDo.of(new BigtableMutationTransform()))
>   
> .apply(BigtableIO.write().withBigtableOptions(bigtableOptionsBuilder).withTableId("table_id"));
> p.run();
> Let me know if you need more context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3198: Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to ...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3198

Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam update-major-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3198


commit 4cee177c8a798015189c1be5061164b22c3ccfe3
Author: Charles Chen 
Date:   2017-05-22T20:03:38Z

Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3197: [BEAM-1999] Remove resolved todo from value provide...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3197

[BEAM-1999] Remove resolved todo from value providers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @mariapython @aaltay PTAL


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1999-remove-old-comment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3197


commit 82c337b4486444eee9b9b76b1a7001b6481808ca
Author: Sourabh Bajaj 
Date:   2017-05-22T20:00:52Z

[BEAM-1999] Remove resolved todo from value providers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020087#comment-16020087
 ] 

Daniel Halperin commented on BEAM-2326:
---

[~kenn], maybe you should take a look at this one?

> Verbose INFO logging with stateful DoFns and Dataflow 
> --
>
> Key: BEAM-2326
> URL: https://issues.apache.org/jira/browse/BEAM-2326
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>  Labels: log
>
> I am seeing a lot of INFO level logging:
> {code}
>  jsonPayload: {
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" 
>   message: "Processing timers for key {} for stateful DoFn"
>  }
>  jsonPayload: {
>   message: "Processing key KV{one of my keys} for stateful DoFn"
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn"  
>   
> }
> {code}
> out of one of my stateful DoFn. There is one such group of logs for each key 
> I process, which leads to a very large amount of logs and possibly to a 
> significant slowdown.
> Also, not sure if the {{Processing timers}} log message is missing some 
> string interpolation or if the empty key is on purpose.
> At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given 
> the large volume.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-2326:
-

Assignee: Kenneth Knowles  (was: Daniel Halperin)

> Verbose INFO logging with stateful DoFns and Dataflow 
> --
>
> Key: BEAM-2326
> URL: https://issues.apache.org/jira/browse/BEAM-2326
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>  Labels: log
>
> I am seeing a lot of INFO level logging:
> {code}
>  jsonPayload: {
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" 
>   message: "Processing timers for key {} for stateful DoFn"
>  }
>  jsonPayload: {
>   message: "Processing key KV{one of my keys} for stateful DoFn"
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn"  
>   
> }
> {code}
> out of one of my stateful DoFn. There is one such group of logs for each key 
> I process, which leads to a very large amount of logs and possibly to a 
> significant slowdown.
> Also, not sure if the {{Processing timers}} log message is missing some 
> string interpolation or if the empty key is on purpose.
> At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given 
> the large volume.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3878

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove Unused DoFnInfo methods

[tgroh] Extract the Main Input PCollection in ParDos

--
[...truncated 1.16 MB...]
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
2017-05-22T19:59:11.933 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.pom
2017-05-22T19:59:11.960 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.pom
 (3 KB at 80.4 KB/sec)
2017-05-22T19:59:11.962 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/derby/derby-project/10.12.1.1/derby-project-10.12.1.1.pom
2017-05-22T19:59:11.988 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/derby/derby-project/10.12.1.1/derby-project-10.12.1.1.pom
 (6 KB at 223.3 KB/sec)
2017-05-22T19:59:11.991 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/derby/derbyclient/10.12.1.1/derbyclient-10.12.1.1.pom
2017-05-22T19:59:12.020 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/derby/derbyclient/10.12.1.1/derbyclient-10.12.1.1.pom
 (3 KB at 74.4 KB/sec)
2017-05-22T19:59:12.023 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/derby/derbynet/10.12.1.1/derbynet-10.12.1.1.pom
2017-05-22T19:59:12.050 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/derby/derbynet/10.12.1.1/derbynet-10.12.1.1.pom
 (3 KB at 90.0 KB/sec)
2017-05-22T19:59:12.053 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/postgresql/postgresql/9.4.1212.jre7/postgresql-9.4.1212.jre7.pom
2017-05-22T19:59:12.081 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/postgresql/postgresql/9.4.1212.jre7/postgresql-9.4.1212.jre7.pom
 (5 KB at 177.7 KB/sec)
2017-05-22T19:59:12.083 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-core-prevjre/1.1.2/pgjdbc-core-prevjre-1.1.2.pom
2017-05-22T19:59:12.110 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-core-prevjre/1.1.2/pgjdbc-core-prevjre-1.1.2.pom
 (7 KB at 254.3 KB/sec)
2017-05-22T19:59:12.112 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-core-parent/1.1.2/pgjdbc-core-parent-1.1.2.pom
2017-05-22T19:59:12.139 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-core-parent/1.1.2/pgjdbc-core-parent-1.1.2.pom
 (16 KB at 581.1 KB/sec)
2017-05-22T19:59:12.141 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-versions/1.1.2/pgjdbc-versions-1.1.2.pom
2017-05-22T19:59:12.170 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/postgresql/pgjdbc-versions/1.1.2/pgjdbc-versions-1.1.2.pom
 (23 KB at 780.6 KB/sec)
[JENKINS] Archiving disabled
2017-05-22T19:59:15.902 [INFO]  
   
2017-05-22T19:59:15.902 [INFO] 

2017-05-22T19:59:15.902 [INFO] Skipping Apache Beam :: Parent
2017-05-22T19:59:15.902 [INFO] This project has been banned from the build due 
to previous failures.
2017-05-22T19:59:15.902 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-05-22T19:59:35.397 [INFO] 

2017-05-22T19:59:35.397 [INFO] Reactor Summary:
2017-05-22T19:59:35.397 [INFO] 
2017-05-22T19:59:35.397 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 

[jira] [Commented] (BEAM-2321) gRPC configuration failure using DataflowRunner and Bigtable

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020085#comment-16020085
 ] 

Daniel Halperin commented on BEAM-2321:
---

[~nkilmer] Which release of gRPC, protobuf, and netty-tcnative are you using? 
Please ensure you're using exactly the released versions in the Apache Beam 
pom.xml file.

> gRPC configuration failure using DataflowRunner and Bigtable
> 
>
> Key: BEAM-2321
> URL: https://issues.apache.org/jira/browse/BEAM-2321
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0
>Reporter: Nigel Kilmer
>Assignee: Daniel Halperin
>
> I'm attempting to run a pipeline that uses the DataflowRunner and writes to 
> Bigtable (v0.9.6.2). This exception is thrown (looks like it's when the 
> BigtableSession is being created):
> java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly 
> configured.
>   at 
> io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139)
>   at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.createSslContext(BigtableSession.java:124)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.access$000(BigtableSession.java:81)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession$2.run(BigtableSession.java:151)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I can run the same pipeline without issue using the DirectRunner instead, and 
> it was also working for me using the 0.7.0 snapshot of Beam last week. I've 
> already checked with the cloud-bigtable-client project; they said that it 
> should be working since I have a dependency on netty_tcnative configured. The 
> fact that the same pipeline works with the DirectRunner and not with the 
> DataflowRunner makes me think it's a DataflowRunner bug.
> My pipeline is pretty simple; it looks like this:
> Pipeline p = Pipeline.create(gcpOptions);
> p.apply(PubsubIO.readProtos(TestProto.class)
> .fromSubscription(pubsubSubscription))
>   .apply(ParDo.of(new BigtableMutationTransform()))
>   
> .apply(BigtableIO.write().withBigtableOptions(bigtableOptionsBuilder).withTableId("table_id"));
> p.run();
> Let me know if you need more context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)