[jira] [Commented] (BEAM-1033) WindowedWordCountIT is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711129#comment-15711129
 ] 

ASF GitHub Bot commented on BEAM-1033:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1479

[BEAM-1033] Bigquery Verifier Retry When Query Failed

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - Add retry when query job failed.
 - check QueryResponse.jobComplete instead of rows. reference from 
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#response

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
Add-retry-when-bq-response-is-failed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1479


commit 6c5a07f1949146d3fc4aed7fa1cfc55863fcdfb2
Author: Mark Liu 
Date:   2016-12-01T06:20:12Z

[BEAM-1033] Bigquery Verifier Retry When Query Failed




> WindowedWordCountIT is flaky
> 
>
> Key: BEAM-1033
> URL: https://issues.apache.org/jira/browse/BEAM-1033
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Mark Liu
>
> Jenkins link:
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console
> Running org.apache.beam.examples.WindowedWordCountIT
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec 
> <<< FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
> testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)  
> Time elapsed: 304.282 sec  <<< FAILURE!
> java.lang.AssertionError: 
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
>  but: Invalid BigQuery response: 
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at 

[GitHub] incubator-beam pull request #1479: [BEAM-1033] Bigquery Verifier Retry When ...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1479

[BEAM-1033] Bigquery Verifier Retry When Query Failed

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - Add retry when query job failed.
 - check QueryResponse.jobComplete instead of rows. reference from 
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#response

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
Add-retry-when-bq-response-is-failed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1479


commit 6c5a07f1949146d3fc4aed7fa1cfc55863fcdfb2
Author: Mark Liu 
Date:   2016-12-01T06:20:12Z

[BEAM-1033] Bigquery Verifier Retry When Query Failed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #1955

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #1955

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710876#comment-15710876
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1478

[BEAM-362] Moved KeyedWorkItem and related classes to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha would you take a look?

It is just a move of a utility class that I think has no place in the 
user-facing SDK. After an initial "LGTM" I will need to do a little dance with 
the Dataflow runner for integration tests. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam KeyedWorkItem

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1478.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1478


commit fb0768a943e2586532965d5593ccf8cf6ae11781
Author: Kenneth Knowles 
Date:   2016-12-01T04:46:04Z

Moved KeyedWorkItem and related classes to runners-core




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1478: [BEAM-362] Moved KeyedWorkItem and relate...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1478

[BEAM-362] Moved KeyedWorkItem and related classes to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha would you take a look?

It is just a move of a utility class that I think has no place in the 
user-facing SDK. After an initial "LGTM" I will need to do a little dance with 
the Dataflow runner for integration tests. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam KeyedWorkItem

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1478.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1478


commit fb0768a943e2586532965d5593ccf8cf6ae11781
Author: Kenneth Knowles 
Date:   2016-12-01T04:46:04Z

Moved KeyedWorkItem and related classes to runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1477: [BEAM-27] Add timerId to TimerData

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1477

[BEAM-27] Add timerId to TimerData

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This `timerId` is generated to be identical to historical behavior, and to 
be unique per time domain and timestamp.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam timerId

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1477


commit d53275d3075d1491d8718837b00093be09f93354
Author: Kenneth Knowles 
Date:   2016-11-23T22:30:57Z

Add timerId to TimerData

This timerId is generated to be identical to historical behavior, and
to be unique per time domain and timestamp.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710850#comment-15710850
 ] 

ASF GitHub Bot commented on BEAM-27:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1477

[BEAM-27] Add timerId to TimerData

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This `timerId` is generated to be identical to historical behavior, and to 
be unique per time domain and timestamp.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam timerId

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1477


commit d53275d3075d1491d8718837b00093be09f93354
Author: Kenneth Knowles 
Date:   2016-11-23T22:30:57Z

Add timerId to TimerData

This timerId is generated to be identical to historical behavior, and
to be unique per time domain and timestamp.




> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1065) FileBasedSource: replace SeekableByteChannel with open(spec, startingPosition)

[Pei He (JIRA)]

Pei He created BEAM-1065:


 Summary: FileBasedSource: replace SeekableByteChannel with 
open(spec, startingPosition)
 Key: BEAM-1065
 URL: https://issues.apache.org/jira/browse/BEAM-1065
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Pei He
Assignee: Pei He


FileBasedReader should be able to open the file with the 
Source.getStartOffset(), and then read forward to find the first input element.

The benefits are:
1. It is easier to implement a ReadableByteChannel.
2. Dynamically splitting won't require file systems to support seeking.
3. Doesn't need to seek to position twice, which is what current API does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1476: auth: add application default credentials...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1476

auth: add application default credentials as fallback

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_auth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1476


commit 191cce8b65b7f66442c5953bfadf9b5c2b20ce04
Author: Vikas Kedigehalli 
Date:   2016-12-01T01:55:20Z

auth: add application default credentials as fallback




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #1954

[Apache Jenkins Server]

See 

--
[...truncated 12238 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.12/slf4j-parent-1.7.12.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.12/slf4j-parent-1.7.12.pom
 (12 KB at 413.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.pom
 (3 KB at 98.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.pom
 (6 KB at 188.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-plexus/1.4.3.2/guice-plexus-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-plexus/1.4.3.2/guice-plexus-1.4.3.2.pom
 (3 KB at 109.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-bean/1.4.3.2/guice-bean-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-bean/1.4.3.2/guice-bean-1.4.3.2.pom
 (3 KB at 88.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject/1.4.3.2/sisu-inject-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject/1.4.3.2/sisu-inject-1.4.3.2.pom
 (2 KB at 43.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-parent/1.4.3.2/sisu-parent-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-parent/1.4.3.2/sisu-parent-1.4.3.2.pom
 (8 KB at 281.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.pom
 (6 KB at 191.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2.pom
 (8 KB at 265.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/2.9.2/guice-parent-2.9.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/2.9.2/guice-parent-2.9.2.pom
 (12 KB at 427.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.pom
 (2 KB at 64.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.1/jackson-annotations-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
 (40 KB at 1074.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
 (315 KB at 6695.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
 (253 KB at 5266.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.1/jackson-annotations-2.6.1.jar
 (46 KB at 936.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
 (711 KB at 10765.4 KB/sec)
[INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #1954

[Apache Jenkins Server]

See 

--
[...truncated 12239 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.12/slf4j-parent-1.7.12.pom
 (12 KB at 413.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.pom
 (3 KB at 98.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.pom
 (6 KB at 188.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-plexus/1.4.3.2/guice-plexus-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-plexus/1.4.3.2/guice-plexus-1.4.3.2.pom
 (3 KB at 109.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-bean/1.4.3.2/guice-bean-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-bean/1.4.3.2/guice-bean-1.4.3.2.pom
 (3 KB at 88.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject/1.4.3.2/sisu-inject-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject/1.4.3.2/sisu-inject-1.4.3.2.pom
 (2 KB at 43.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-parent/1.4.3.2/sisu-parent-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-parent/1.4.3.2/sisu-parent-1.4.3.2.pom
 (8 KB at 281.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.pom
 (6 KB at 191.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2.pom
 (8 KB at 265.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/2.9.2/guice-parent-2.9.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/inject/guice-parent/2.9.2/guice-parent-2.9.2.pom
 (12 KB at 427.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.pom
 (2 KB at 64.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.1/jackson-annotations-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
 (40 KB at 1074.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
 (315 KB at 6695.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
 (253 KB at 5266.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.1/jackson-annotations-2.6.1.jar
 (46 KB at 936.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
 (711 KB at 10765.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.jar
[INFO] Downloaded: 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #1693

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #1693

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1064) Convert Jenkins jobs to DSL

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710476#comment-15710476
 ] 

Jason Kuster commented on BEAM-1064:


I've updated all of the jobs based on the pull request and am tracking down any 
failures. So far, the Coveralls token got trampled. Re-adding it, and updating 
the PR to fix the token getting removed.

> Convert Jenkins jobs to DSL
> ---
>
> Key: BEAM-1064
> URL: https://issues.apache.org/jira/browse/BEAM-1064
> Project: Beam
>  Issue Type: Improvement
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>  Labels: jenkins
>
> Move Jenkins jobs to DSL. PR is here:
> https://github.com/apache/incubator-beam/pull/1390



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1064) Convert Jenkins jobs to DSL

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-1064:
--

 Summary: Convert Jenkins jobs to DSL
 Key: BEAM-1064
 URL: https://issues.apache.org/jira/browse/BEAM-1064
 Project: Beam
  Issue Type: Improvement
Reporter: Jason Kuster
Assignee: Jason Kuster


Move Jenkins jobs to DSL. PR is here:
https://github.com/apache/incubator-beam/pull/1390



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #1953

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Update examples archetype with runner profiles

--
[...truncated 12265 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
 (253 KB at 6018.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
 (40 KB at 924.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
 (315 KB at 6051.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
 (711 KB at 8881.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar 
(258 KB at 3104.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
 (123 KB at 1286.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
 (1140 KB at 10081.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2-no_aop.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.jar
 (219 KB at 1807.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.jar
 (201 KB at 1539.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.jar
 (157 KB at 1195.9 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
 (32 KB at 198.6 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2-no_aop.jar
 (469 KB at 2947.0 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.jar
 (17 KB at 101.4 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
 (2079 KB at 11743.8 KB/sec)
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. FAILURE [  1.432 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  4.014 s]
[INFO] Apache Beam :: SDKs  SUCCESS [ 10.786 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  2.227 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [03:52 min]
[INFO] Apache Beam :: Runners . SUCCESS [  2.003 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [ 43.374 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [01:59 min]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [ 44.093 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  1.449 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS 
[04:58 min]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [ 30.415 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [ 11.317 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [ 18.966 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [ 17.124 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [ 27.261 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [ 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #1953

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Update examples archetype with runner profiles

--
[...truncated 12264 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.1/jackson-annotations-2.6.1.jar
 (46 KB at 1433.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.1/jackson-core-2.6.1.jar
 (253 KB at 6018.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5/httpmime-4.5.jar
 (40 KB at 924.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar
 (315 KB at 6051.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar
 (711 KB at 8881.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar 
(258 KB at 3104.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
 (123 KB at 1286.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.1/jackson-databind-2.6.1.jar
 (1140 KB at 10081.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2-no_aop.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.7/plexus-utils-2.0.7.jar
 (219 KB at 1807.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-plexus/1.4.3.2/sisu-inject-plexus-1.4.3.2.jar
 (201 KB at 1539.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.3.2/sisu-inject-bean-1.4.3.2.jar
 (157 KB at 1195.9 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
 (32 KB at 198.6 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.9.2/sisu-guice-2.9.2-no_aop.jar
 (469 KB at 2947.0 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.7.12/jcl-over-slf4j-1.7.12.jar
 (17 KB at 101.4 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.0.1.201506240215-r/org.eclipse.jgit-4.0.1.201506240215-r.jar
 (2079 KB at 11743.8 KB/sec)
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. FAILURE [  1.432 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  4.014 s]
[INFO] Apache Beam :: SDKs  SUCCESS [ 10.786 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  2.227 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [03:52 min]
[INFO] Apache Beam :: Runners . SUCCESS [  2.003 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [ 43.374 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [01:59 min]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [ 44.093 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  1.449 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS 
[04:58 min]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [ 30.415 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [ 11.317 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [ 18.966 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #1952

[Apache Jenkins Server]

See 

--
[...truncated 12292 lines...]
[INFO] Apache Beam :: Parent .. FAILURE [  1.338 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.367 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  5.950 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  1.632 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [02:37 min]
[INFO] Apache Beam :: Runners . SUCCESS [  1.620 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [ 32.384 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [01:57 min]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [ 23.437 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  2.706 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS 
[04:25 min]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [ 22.792 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [  9.532 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [ 15.845 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [ 17.217 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [ 26.480 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [ 18.708 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [  1.552 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter SUCCESS [ 
14.090 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples SUCCESS [ 
12.115 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [  1.217 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library SUCCESS [ 
14.511 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter  SUCCESS [ 29.234 s]
[INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests  SUCCESS [  8.690 s]
[INFO] Apache Beam :: Runners :: Flink  SUCCESS [  1.336 s]
[INFO] Apache Beam :: Runners :: Flink :: Core  SUCCESS [ 38.778 s]
[INFO] Apache Beam :: Runners :: Flink :: Examples  SUCCESS [ 12.396 s]
[INFO] Apache Beam :: Runners :: Spark  SUCCESS [02:47 min]
[INFO] Apache Beam :: Runners :: Apex . SUCCESS [ 54.895 s]
[INFO] Apache Beam :: Examples  SUCCESS [  1.477 s]
[INFO] Apache Beam :: Examples :: Java  SUCCESS [07:41 min]
[INFO] Apache Beam :: Examples :: Java 8 .. SUCCESS [ 15.316 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 27:18 min
[INFO] Finished at: 2016-12-01T00:55:55+00:00
[INFO] Final Memory: 211M/1014M
[INFO] 
Waiting for Jenkins to finish collecting data
[ERROR] Failed to execute goal 
org.eluder.coveralls:coveralls-maven-plugin:4.1.0:report (default-cli) on 
project beam-parent: Build error: Either repository token or service with job 
id must be defined -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.eluder.coveralls:coveralls-maven-plugin:4.1.0:report (default-cli) on 
project beam-parent: Build error
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven32Launcher.main(Maven32Launcher.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 

[jira] [Updated] (BEAM-1063) WordCount on DirectRunner is hanging for ~1 minute after job runs

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1063:
--
Fix Version/s: (was: 0.4.0-incubating)
   Not applicable

> WordCount on DirectRunner is hanging for ~1 minute after job runs
> -
>
> Key: BEAM-1063
> URL: https://issues.apache.org/jira/browse/BEAM-1063
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>
> generate example archetype
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \   
>  
>  -Dexec.args="--inputFile=pom.xml --output=direct/counts"
> {code}
> In the demo I gave 2 weeks ago (sorry, not sure what commit this was at) this 
> finished quickly.
> In the demo Ben gave yesterday, it hung. But only in the Direct runner. Some 
> regression in that time?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #277

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1465: [BEAM-905] [BEAM-1056] [BEAM-892] Update ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1465


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-905) Archetype pom needs to generalize dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710342#comment-15710342
 ] 

ASF GitHub Bot commented on BEAM-905:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1465


> Archetype pom needs to generalize dependencies
> --
>
> Key: BEAM-905
> URL: https://issues.apache.org/jira/browse/BEAM-905
> Project: Beam
>  Issue Type: Bug
>Affects Versions: 0.4.0-incubating
> Environment: Currently the archetype pom includes the direct runner 
> and the dataflow one, but not the others. It should do the same magic as the 
> main examples.
>Reporter: Frances Perry
>Assignee: Pei He
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Update examples archetype with runner profiles

[dhalperi]

Update examples archetype with runner profiles

This makes it possible to run the examples on all runners.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/265c7924
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/265c7924
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/265c7924

Branch: refs/heads/master
Commit: 265c79241f802f4d895648c1b1c4b75e6846d245
Parents: a20bc47
Author: Dan Halperin 
Authored: Wed Nov 30 11:10:20 2016 -0800
Committer: Dan Halperin 
Committed: Wed Nov 30 16:28:19 2016 -0800

--
 .../main/resources/archetype-resources/pom.xml  | 123 ---
 1 file changed, 103 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/265c7924/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 031ee88..df2e9f3 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -26,6 +26,10 @@
 
   jar
 
+  
+0.4.0-incubating-SNAPSHOT
+  
+
   
 
   apache.snapshots
@@ -85,36 +89,108 @@
 
   
 
-  
-
-
-  org.apache.beam
-  beam-sdks-java-core
-  0.4.0-incubating-SNAPSHOT
-
+  
+
+  direct-runner
+  
+  
+
+  org.apache.beam
+  beam-runners-direct-java
+  ${beam.version}
+  runtime
+
+  
+
 
-
-
-  org.apache.beam
-  beam-runners-direct-java
-  0.4.0-incubating-SNAPSHOT
-  runtime
-
+
+  apex-runner
+  
+  
+
+  org.apache.beam
+  beam-runners-apex
+  ${beam.version}
+  runtime
+
+  
+
+
+
+  dataflow-runner
+  
+  
+
+  org.apache.beam
+  beam-runners-google-cloud-dataflow-java
+  ${beam.version}
+  runtime
+
+  
+
 
+
+  flink-runner
+  
+  
+
+  org.apache.beam
+  beam-runners-flink_2.10
+  ${beam.version}
+  runtime
+
+  
+
+
+
+  spark-runner
+  
+  
+
+  org.apache.beam
+  beam-runners-spark
+  ${beam.version}
+  runtime
+
+
+  org.apache.spark
+  spark-streaming_2.10
+  1.6.2
+  runtime
+  
+
+  org.slf4j
+  jul-to-slf4j
+
+  
+
+
+  com.fasterxml.jackson.module
+  jackson-module-scala_2.10
+  2.7.2
+  runtime
+
+  
+
+  
+
+  
+
 
   org.apache.beam
-  beam-runners-google-cloud-dataflow-java
-  0.4.0-incubating-SNAPSHOT
-  runtime
+  beam-sdks-java-core
+  ${beam.version}
 
 
-
+
 
   org.apache.beam
   beam-sdks-java-io-google-cloud-platform
-  0.4.0-incubating-SNAPSHOT
+  ${beam.version}
 
 
+
 
   com.google.api-client
   google-api-client
@@ -129,7 +205,6 @@
   
 
 
-
 
   com.google.apis
   google-api-services-bigquery
@@ -212,5 +287,13 @@
   junit
   4.11
 
+
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  ${beam.version}
+  test
+
   
 

[1/2] incubator-beam git commit: Closes #1465

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master a20bc4793 -> 711c68092


Closes #1465


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/711c6809
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/711c6809
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/711c6809

Branch: refs/heads/master
Commit: 711c68092fd771c3f9be4a5d0dd0ecf077f1aeab
Parents: a20bc47 265c792
Author: Dan Halperin 
Authored: Wed Nov 30 16:28:19 2016 -0800
Committer: Dan Halperin 
Committed: Wed Nov 30 16:28:19 2016 -0800

--
 .../main/resources/archetype-resources/pom.xml  | 123 ---
 1 file changed, 103 insertions(+), 20 deletions(-)
--

[GitHub] incubator-beam pull request #1471: Preserves compressed windows in PushbackS...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1471


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Preserves compressed windows in PushbackSideInputDoFnRunner

[tgroh]

Preserves compressed windows in PushbackSideInputDoFnRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/38f0b11c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/38f0b11c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/38f0b11c

Branch: refs/heads/master
Commit: 38f0b11cc9028cf347e3c96b6e6116e5a5a9972d
Parents: 565e99f
Author: Eugene Kirpichov 
Authored: Wed Nov 30 14:28:51 2016 -0800
Committer: Thomas Groh 
Committed: Wed Nov 30 16:26:33 2016 -0800

--
 .../core/PushbackSideInputDoFnRunner.java   | 20 
 .../core/PushbackSideInputDoFnRunnerTest.java   | 18 +++---
 2 files changed, 27 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/38f0b11c/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
index 8c169da..460154d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
@@ -74,17 +74,29 @@ public class PushbackSideInputDoFnRunner 
implements DoFnRunner<
   processElement(elem);
   return Collections.emptyList();
 }
-ImmutableList.Builder pushedBack = 
ImmutableList.builder();
+ImmutableList.Builder readyWindowsBuilder = 
ImmutableList.builder();
+ImmutableList.Builder pushedBackWindowsBuilder = 
ImmutableList.builder();
 for (WindowedValue windowElem : elem.explodeWindows()) {
   BoundedWindow mainInputWindow = 
Iterables.getOnlyElement(windowElem.getWindows());
   if (isReady(mainInputWindow)) {
-processElement(windowElem);
+readyWindowsBuilder.add(mainInputWindow);
   } else {
 notReadyWindows.add(mainInputWindow);
-pushedBack.add(windowElem);
+pushedBackWindowsBuilder.add(mainInputWindow);
   }
 }
-return pushedBack.build();
+ImmutableList readyWindows = readyWindowsBuilder.build();
+ImmutableList pushedBackWindows = 
pushedBackWindowsBuilder.build();
+if (!readyWindows.isEmpty()) {
+  processElement(
+  WindowedValue.of(
+  elem.getValue(), elem.getTimestamp(), readyWindows, 
elem.getPane()));
+}
+return pushedBackWindows.isEmpty()
+? ImmutableList.of()
+: ImmutableList.of(
+WindowedValue.of(
+elem.getValue(), elem.getTimestamp(), pushedBackWindows, 
elem.getPane()));
   }
 
   private boolean isReady(BoundedWindow mainInputWindow) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/38f0b11c/runners/core-java/src/test/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunnerTest.java
index 59a7c92..f8f4604 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunnerTest.java
@@ -17,9 +17,9 @@
  */
 package org.apache.beam.runners.core;
 
+import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
 import static org.junit.Assert.assertThat;
 import static org.mockito.Mockito.when;
@@ -27,7 +27,6 @@ import static org.mockito.Mockito.when;
 import com.google.common.collect.ImmutableList;
 import java.util.ArrayList;
 import java.util.List;
-
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.Sum;
@@ -131,7 +130,7 @@ public class PushbackSideInputDoFnRunnerTest {
 PaneInfo.ON_TIME_AND_ONLY_FIRING);
 Iterable multiWindowPushback =
 runner.processElementInReadyWindows(multiWindow);
-assertThat(multiWindowPushback, equalTo(multiWindow.explodeWindows()));
+assertThat(multiWindowPushback, contains(multiWindow));
 assertThat(underlying.inputElems, 
Matchers.emptyIterable());
   }
 

[1/2] incubator-beam git commit: This closes #1471

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 565e99fbf -> a20bc4793


This closes #1471


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a20bc479
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a20bc479
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a20bc479

Branch: refs/heads/master
Commit: a20bc479347b2a10307ada45dc7220ee00671fac
Parents: 565e99f 38f0b11
Author: Thomas Groh 
Authored: Wed Nov 30 16:26:33 2016 -0800
Committer: Thomas Groh 
Committed: Wed Nov 30 16:26:33 2016 -0800

--
 .../core/PushbackSideInputDoFnRunner.java   | 20 
 .../core/PushbackSideInputDoFnRunnerTest.java   | 18 +++---
 2 files changed, 27 insertions(+), 11 deletions(-)
--

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #275

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1063) WordCount on DirectRunner is hanging for ~1 minute after job runs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710287#comment-15710287
 ] 

ASF GitHub Bot commented on BEAM-1063:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1470


> WordCount on DirectRunner is hanging for ~1 minute after job runs
> -
>
> Key: BEAM-1063
> URL: https://issues.apache.org/jira/browse/BEAM-1063
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
>
> generate example archetype
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \   
>  
>  -Dexec.args="--inputFile=pom.xml --output=direct/counts"
> {code}
> In the demo I gave 2 weeks ago (sorry, not sure what commit this was at) this 
> finished quickly.
> In the demo Ben gave yesterday, it hung. But only in the Direct runner. Some 
> regression in that time?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Shutdown DynamicSplit Executor in Cleanup

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c8f2cdb22 -> 565e99fbf


Shutdown DynamicSplit Executor in Cleanup

This ensures that the threads will be shut off when the pipeline shuts
down, enabling a JVM with no more work to do to shut down as well.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6ef9a288
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6ef9a288
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6ef9a288

Branch: refs/heads/master
Commit: 6ef9a288e281a423905c2cba520274d1c4e4747b
Parents: c8f2cdb
Author: Thomas Groh 
Authored: Wed Nov 30 14:30:14 2016 -0800
Committer: Dan Halperin 
Committed: Wed Nov 30 16:08:20 2016 -0800

--
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java   | 6 --
 .../beam/runners/direct/BoundedReadEvaluatorFactoryTest.java   | 6 ++
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ef9a288/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 65b622f..8874a04 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -58,7 +58,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
*/
   private static final long REQUIRED_DYNAMIC_SPLIT_ORIGINAL_SIZE = 0;
   private final EvaluationContext evaluationContext;
-  private final ExecutorService executor = Executors.newCachedThreadPool();
+  @VisibleForTesting final ExecutorService executor = 
Executors.newCachedThreadPool();
 
   private final long minimumDynamicSplitSize;
 
@@ -87,7 +87,9 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
   }
 
   @Override
-  public void cleanup() {}
+  public void cleanup() {
+executor.shutdown();
+  }
 
   /**
* A {@link BoundedReadEvaluator} produces elements from an underlying 
{@link BoundedSource},

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ef9a288/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
index dee95a7..b1ff689 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
@@ -335,6 +335,12 @@ public class BoundedReadEvaluatorFactoryTest {
 assertThat(TestSource.readerClosed, is(true));
   }
 
+  @Test
+  public void cleanupShutsDownExecutor() {
+factory.cleanup();
+assertThat(factory.executor.isShutdown(), is(true));
+  }
+
   private static class TestSource extends OffsetBasedSource {
 private static boolean readerClosed;
 private final Coder coder;

[GitHub] incubator-beam pull request #1470: [BEAM-1063] Shutdown DynamicSplit Executo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1470

[dhalperi]

Closes #1470


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/565e99fb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/565e99fb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/565e99fb

Branch: refs/heads/master
Commit: 565e99fbf8f7a9e9863bdfcfb514e2098365bbc6
Parents: c8f2cdb 6ef9a28
Author: Dan Halperin 
Authored: Wed Nov 30 16:08:22 2016 -0800
Committer: Dan Halperin 
Committed: Wed Nov 30 16:08:22 2016 -0800

--
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java   | 6 --
 .../beam/runners/direct/BoundedReadEvaluatorFactoryTest.java   | 6 ++
 2 files changed, 10 insertions(+), 2 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #1691

[Apache Jenkins Server]

See 


--
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/incubator-beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/${sha1}^{commit} # timeout=10
 > git rev-parse ${sha1}^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/${sha1}^{commit} # timeout=10
 > git rev-parse ${sha1}^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/${sha1}^{commit} # timeout=10
 > git rev-parse ${sha1}^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Sending e-mails to: commits@beam.incubator.apache.org k...@google.com

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710197#comment-15710197
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1475

[BEAM-551] Add TextIO.Write support for ValueProvider

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/textio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1475


commit 18d3ee531c45bf972ca5da701ede0ad43c4760f7
Author: Sam McVeety 
Date:   2016-11-30T22:06:59Z

Add TextIO.Write support




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1475: [BEAM-551] Add TextIO.Write support for V...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1475

[BEAM-551] Add TextIO.Write support for ValueProvider

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/textio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1475


commit 18d3ee531c45bf972ca5da701ede0ad43c4760f7
Author: Sam McVeety 
Date:   2016-11-30T22:06:59Z

Add TextIO.Write support




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Revert "Improvements to ReduceFnRunner prefetching"

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4b682039d -> c8f2cdb22


Revert "Improvements to ReduceFnRunner prefetching"

This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which
contained some incompatibilities outside of runners-core.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/aaa3b91e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/aaa3b91e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/aaa3b91e

Branch: refs/heads/master
Commit: aaa3b91e1e7b39dd585314a6017235cdd127e923
Parents: 4b68203
Author: Kenneth Knowles 
Authored: Wed Nov 30 15:21:53 2016 -0800
Committer: Kenneth Knowles 
Committed: Wed Nov 30 15:21:53 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 -
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 488 +++
 .../apache/beam/runners/core/WatermarkHold.java |   5 -
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |   4 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   6 +-
 .../sdk/util/state/InMemoryTimerInternals.java  |  22 +-
 .../beam/sdk/util/state/TimerCallback.java  |   9 +-
 .../util/state/InMemoryTimerInternalsTest.java  |  54 +-
 12 files changed, 224 insertions(+), 407 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aaa3b91e/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
index 294f21d..8b10813 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
@@ -27,6 +27,7 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.KeyedWorkItem;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.TimerInternals;
+import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
@@ -72,9 +73,9 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 
   @Override
   public void processElement(ProcessContext c) throws Exception {
-KeyedWorkItem keyedWorkItem = c.element();
+KeyedWorkItem element = c.element();
 
-K key = keyedWorkItem.key();
+K key = c.element().key();
 TimerInternals timerInternals = c.windowingInternals().timerInternals();
 StateInternals stateInternals = 
stateInternalsFactory.stateInternalsForKey(key);
 
@@ -92,8 +93,10 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 reduceFn,
 c.getPipelineOptions());
 
-reduceFnRunner.processElements(keyedWorkItem.elementsIterable());
-reduceFnRunner.onTimers(keyedWorkItem.timersIterable());
+reduceFnRunner.processElements(element.elementsIterable());
+for (TimerData timer : element.timersIterable()) {
+  reduceFnRunner.onTimer(timer);
+}
 reduceFnRunner.persist();
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aaa3b91e/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
index 69a4cfd..8140243 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
@@ -54,10 +54,6 @@ public class PaneInfoTracker {
 state.access(PANE_INFO_TAG).clear();
   }
 
-  public void prefetchPaneInfo(ReduceFn.Context context) {
-context.state().access(PaneInfoTracker.PANE_INFO_TAG).readLater();
-  }
-
   /**
* Return a ({@link ReadableState} for) the pane info appropriate for {@code 
context}. The pane
* info includes the timing for the pane, who's calculation is quite subtle.

[jira] [Commented] (BEAM-986) ReduceFnRunner doesn't batch prefetching pane firings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710193#comment-15710193
 ] 

ASF GitHub Bot commented on BEAM-986:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1474


> ReduceFnRunner doesn't batch prefetching pane firings
> -
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is 
> processed sequentially with sequential state fetches instead of a bulk 
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if 
> multiple timers are fired at once each is processed sequentially without 
> batched prefetching



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1474

[tgroh]

This closes #1474


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c8f2cdb2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c8f2cdb2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c8f2cdb2

Branch: refs/heads/master
Commit: c8f2cdb223cf39d76e733e4dc3fffb5487d41fa4
Parents: 4b68203 aaa3b91
Author: Thomas Groh 
Authored: Wed Nov 30 15:30:30 2016 -0800
Committer: Thomas Groh 
Committed: Wed Nov 30 15:30:30 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 -
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 488 +++
 .../apache/beam/runners/core/WatermarkHold.java |   5 -
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |   4 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   6 +-
 .../sdk/util/state/InMemoryTimerInternals.java  |  22 +-
 .../beam/sdk/util/state/TimerCallback.java  |   9 +-
 .../util/state/InMemoryTimerInternalsTest.java  |  54 +-
 12 files changed, 224 insertions(+), 407 deletions(-)
--

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1690

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-986) ReduceFnRunner doesn't batch prefetching pane firings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710185#comment-15710185
 ] 

ASF GitHub Bot commented on BEAM-986:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1474

[BEAM-986] Revert "Improvements to ReduceFnRunner prefetching"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which 
contained some incompatibilities outside of runners-core. The PR is #1366, for 
reference.

R: anyone, pinging @tgroh or @dhalperi 
CC: @scwhittle 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam rollback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1474


commit aaa3b91e1e7b39dd585314a6017235cdd127e923
Author: Kenneth Knowles 
Date:   2016-11-30T23:21:53Z

Revert "Improvements to ReduceFnRunner prefetching"

This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which
contained some incompatibilities outside of runners-core.




> ReduceFnRunner doesn't batch prefetching pane firings
> -
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is 
> processed sequentially with sequential state fetches instead of a bulk 
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if 
> multiple timers are fired at once each is processed sequentially without 
> batched prefetching



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1949

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1474: [BEAM-986] Revert "Improvements to Reduce...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1474

[BEAM-986] Revert "Improvements to ReduceFnRunner prefetching"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which 
contained some incompatibilities outside of runners-core. The PR is #1366, for 
reference.

R: anyone, pinging @tgroh or @dhalperi 
CC: @scwhittle 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam rollback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1474


commit aaa3b91e1e7b39dd585314a6017235cdd127e923
Author: Kenneth Knowles 
Date:   2016-11-30T23:21:53Z

Revert "Improvements to ReduceFnRunner prefetching"

This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which
contained some incompatibilities outside of runners-core.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1473: Adds TransformEvaluator.startBundle()

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1473

Adds TransformEvaluator.startBundle()

For symmetry with finishBundle() and to allow decoupling creation of
a ParDoEvaluator from the time it calls fn.startBundle().

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam start-bundle

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1473


commit a3c2e1c755ae51d6ad7058b7a811006ab7455ace
Author: Eugene Kirpichov 
Date:   2016-11-30T23:23:26Z

Adds TransformEvaluator.startBundle()

For symmetry with finishBundle() and to allow decoupling creation of
a ParDoEvaluator from the time it calls fn.startBundle().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Jenkins build is still unstable: beam_PostCommit_MavenVerify #1948

[Kenneth Knowles]

This is a Dataflow-specific linking error. I am investigating and
proceeding with a temporary rollback.

On Wed, Nov 30, 2016 at 3:06 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
>

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1948

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1472: Add snippet for standard sql

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1472

Add snippet for standard sql

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @hadarhg PTAL 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam BEAM-update-bq-snippets

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1472


commit abe0c5785823cef21b5bfc3935ca4ab2c5827a5a
Author: Sourabh Bajaj 
Date:   2016-11-30T22:59:15Z

Add snippet for standard sql




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1471: Preserves compressed windows in PushbackS...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1471

Preserves compressed windows in PushbackSideInputDoFnRunner

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam pushback-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1471






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1388

[kenn]

This closes #1388


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4b682039
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4b682039
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4b682039

Branch: refs/heads/master
Commit: 4b682039d1286861bb2658401913eddf8bdbfb4a
Parents: 4a7da91 68a2025
Author: Kenneth Knowles 
Authored: Wed Nov 30 14:25:31 2016 -0800
Committer: Kenneth Knowles 
Committed: Wed Nov 30 14:25:31 2016 -0800

--
 .../runners/core/PerKeyCombineFnRunners.java| 263 +++
 .../FlinkMergingNonShuffleReduceFunction.java   |   2 +-
 .../FlinkMergingPartialReduceFunction.java  |   2 +-
 .../functions/FlinkMergingReduceFunction.java   |   2 +-
 .../functions/FlinkPartialReduceFunction.java   |   2 +-
 .../functions/FlinkReduceFunction.java  |   2 +-
 .../org/apache/beam/sdk/transforms/Combine.java |  37 ++-
 .../beam/sdk/util/PerKeyCombineFnRunners.java   | 258 --
 8 files changed, 297 insertions(+), 271 deletions(-)
--

[GitHub] incubator-beam pull request #1388: Move PerKeyCombineFnRunners to runners/co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1388


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1470: [BEAM-1063] Shutdown DynamicSplit Executo...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1470

[BEAM-1063] Shutdown DynamicSplit Executor in Cleanup

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This ensures that the threads will be shut off when the pipeline shuts
down, enabling a JVM with no more work to do to shut down as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
direct_runner_finishes_and_hangs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1470.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1470


commit 2780181991afd62093ca79330f95ff7c759c2f6b
Author: Thomas Groh 
Date:   2016-11-30T22:30:14Z

Shutdown DynamicSplit Executor in Cleanup

This ensures that the threads will be shut off when the pipeline shuts
down, enabling a JVM with no more work to do to shut down as well.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1063) WordCount on DirectRunner is hanging for ~1 minute after job runs

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1063:
-

 Summary: WordCount on DirectRunner is hanging for ~1 minute after 
job runs
 Key: BEAM-1063
 URL: https://issues.apache.org/jira/browse/BEAM-1063
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Thomas Groh


generate example archetype

{code}
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--inputFile=pom.xml --output=direct/counts"
{code}

In the demo I gave 2 weeks ago (sorry, not sure what commit this was at) this 
finished quickly.

In the demo Ben gave yesterday, it hung. But only in the Direct runner. Some 
regression in that time?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-641) Need to test the generated archetypes projects

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710020#comment-15710020
 ] 

ASF GitHub Bot commented on BEAM-641:
-

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1057


> Need to test the generated archetypes projects
> --
>
> Key: BEAM-641
> URL: https://issues.apache.org/jira/browse/BEAM-641
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Critical
> Fix For: 0.4.0-incubating
>
>
> Travis and Jenkins pre-submits don't test building the generated archetypes 
> projects.
> Currently, changes to archetypes have to be manually verified by:
> mvn archetype:generate \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeVersion=0.3.0-incubating-SNAPSHOT \
> -DgroupId=com.example \
> -DartifactId=first-beam \
> -Dversion="0.3.0-incubating-SNAPSHOT" \
> -DinteractiveMode=false \
> -Dpackage=org.apache.beam.examples
> and did "mvn clean install" in first-beam project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-986) ReduceFnRunner doesn't batch prefetching pane firings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709991#comment-15709991
 ] 

ASF GitHub Bot commented on BEAM-986:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1366


> ReduceFnRunner doesn't batch prefetching pane firings
> -
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is 
> processed sequentially with sequential state fetches instead of a bulk 
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if 
> multiple timers are fired at once each is processed sequentially without 
> batched prefetching



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1057: [BEAM-641] Travis: verify the generated e...

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1057


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Improvements to ReduceFnRunner prefetching

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b75a76459 -> 4a7da91f0


Improvements to ReduceFnRunner prefetching

- add prefetch* methods for prefetching state matching existing methods
- replace onTimer with batched onTimers method to allow prefetching
  across timers
- prefetch triggers in processElements


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4282c67c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4282c67c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4282c67c

Branch: refs/heads/master
Commit: 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b
Parents: b75a764
Author: Sam Whittle 
Authored: Thu Nov 10 12:59:49 2016 -0800
Committer: Kenneth Knowles 
Committed: Wed Nov 30 13:32:05 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 488 ---
 .../apache/beam/runners/core/WatermarkHold.java |   5 +
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |   4 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   6 +-
 .../sdk/util/state/InMemoryTimerInternals.java  |  22 +-
 .../beam/sdk/util/state/TimerCallback.java  |   9 +-
 .../util/state/InMemoryTimerInternalsTest.java  |  54 +-
 12 files changed, 407 insertions(+), 224 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4282c67c/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
index 8b10813..294f21d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
@@ -27,7 +27,6 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.KeyedWorkItem;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.TimerInternals;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
@@ -73,9 +72,9 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 
   @Override
   public void processElement(ProcessContext c) throws Exception {
-KeyedWorkItem element = c.element();
+KeyedWorkItem keyedWorkItem = c.element();
 
-K key = c.element().key();
+K key = keyedWorkItem.key();
 TimerInternals timerInternals = c.windowingInternals().timerInternals();
 StateInternals stateInternals = 
stateInternalsFactory.stateInternalsForKey(key);
 
@@ -93,10 +92,8 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 reduceFn,
 c.getPipelineOptions());
 
-reduceFnRunner.processElements(element.elementsIterable());
-for (TimerData timer : element.timersIterable()) {
-  reduceFnRunner.onTimer(timer);
-}
+reduceFnRunner.processElements(keyedWorkItem.elementsIterable());
+reduceFnRunner.onTimers(keyedWorkItem.timersIterable());
 reduceFnRunner.persist();
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4282c67c/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
index 8140243..69a4cfd 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
@@ -54,6 +54,10 @@ public class PaneInfoTracker {
 state.access(PANE_INFO_TAG).clear();
   }
 
+  public void prefetchPaneInfo(ReduceFn.Context context) {
+context.state().access(PaneInfoTracker.PANE_INFO_TAG).readLater();
+  }
+
   /**
* Return a ({@link ReadableState} for) the pane info appropriate for {@code 
context}. The pane
* info includes the timing for the pane, who's calculation is quite subtle.

[GitHub] incubator-beam pull request #1366: [BEAM-986] Improve ReduceFnRunner prefetc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1366


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1366

[kenn]

This closes #1366


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4a7da91f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4a7da91f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4a7da91f

Branch: refs/heads/master
Commit: 4a7da91f063c0de07fca250d1016b85832225732
Parents: b75a764 4282c67
Author: Kenneth Knowles 
Authored: Wed Nov 30 13:32:27 2016 -0800
Committer: Kenneth Knowles 
Committed: Wed Nov 30 13:32:27 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 488 ---
 .../apache/beam/runners/core/WatermarkHold.java |   5 +
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |   4 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   6 +-
 .../sdk/util/state/InMemoryTimerInternals.java  |  22 +-
 .../beam/sdk/util/state/TimerCallback.java  |   9 +-
 .../util/state/InMemoryTimerInternalsTest.java  |  54 +-
 12 files changed, 407 insertions(+), 224 deletions(-)
--

[GitHub] incubator-beam pull request #1454: Few datastore related fixes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1454


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1454

[robertwb]

Closes #1454


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/aa9071d5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/aa9071d5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/aa9071d5

Branch: refs/heads/python-sdk
Commit: aa9071d56120a1c0f91cc580fb956570113ee104
Parents: 70c1de9 495a2d8
Author: Robert Bradshaw 
Authored: Wed Nov 30 14:08:58 2016 -0800
Committer: Robert Bradshaw 
Committed: Wed Nov 30 14:08:58 2016 -0800

--
 .../examples/cookbook/datastore_wordcount.py| 210 +++
 .../apache_beam/examples/datastore_wordcount.py | 203 --
 sdks/python/setup.py|   2 +-
 3 files changed, 211 insertions(+), 204 deletions(-)
--

[1/2] incubator-beam git commit: Few datastoreio fixes

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 70c1de9b9 -> aa9071d56


Few datastoreio fixes

  - pin googledatastore version to 6.4.1
  - add num_shards options to datastore wordcount example
  - move datastore wordcount example to cookbook directory


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/495a2d8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/495a2d8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/495a2d8c

Branch: refs/heads/python-sdk
Commit: 495a2d8c01949e4248d7e6dc9ad7a04168a292d1
Parents: 70c1de9
Author: Vikas Kedigehalli 
Authored: Tue Nov 29 10:02:14 2016 -0800
Committer: Vikas Kedigehalli 
Committed: Wed Nov 30 12:18:53 2016 -0800

--
 .../examples/cookbook/datastore_wordcount.py| 210 +++
 .../apache_beam/examples/datastore_wordcount.py | 203 --
 sdks/python/setup.py|   2 +-
 3 files changed, 211 insertions(+), 204 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/495a2d8c/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py 
b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
new file mode 100644
index 000..eb62614
--- /dev/null
+++ b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A word-counting workflow that uses Google Cloud Datastore."""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+import uuid
+
+from google.datastore.v1 import entity_pb2
+from google.datastore.v1 import query_pb2
+from googledatastore import helper as datastore_helper, PropertyFilter
+
+import apache_beam as beam
+from apache_beam.io.datastore.v1.datastoreio import ReadFromDatastore
+from apache_beam.io.datastore.v1.datastoreio import WriteToDatastore
+from apache_beam.utils.options import GoogleCloudOptions
+from apache_beam.utils.options import PipelineOptions
+from apache_beam.utils.options import SetupOptions
+
+empty_line_aggregator = beam.Aggregator('emptyLines')
+average_word_size_aggregator = beam.Aggregator('averageWordLength',
+   beam.combiners.MeanCombineFn(),
+   float)
+
+
+class WordExtractingDoFn(beam.DoFn):
+  """Parse each line of input text into words."""
+
+  def process(self, context):
+"""Returns an iterator over words in contents of Cloud Datastore entity.
+The element is a line of text.  If the line is blank, note that, too.
+Args:
+  context: the call-specific context: data and aggregator.
+Returns:
+  The processed element.
+"""
+content_value = context.element.properties.get('content', None)
+text_line = ''
+if content_value:
+  text_line = content_value.string_value
+
+if not text_line:
+  context.aggregate_to(empty_line_aggregator, 1)
+words = re.findall(r'[A-Za-z\']+', text_line)
+for w in words:
+  context.aggregate_to(average_word_size_aggregator, len(w))
+return words
+
+
+class EntityWrapper(object):
+  """Create a Cloud Datastore entity from the given string."""
+  def __init__(self, namespace, kind, ancestor):
+self._namespace = namespace
+self._kind = kind
+self._ancestor = ancestor
+
+  def make_entity(self, content):
+entity = entity_pb2.Entity()
+if self._namespace is not None:
+  entity.key.partition_id.namespace_id = self._namespace
+
+# All entities created will have the same ancestor
+datastore_helper.add_key_path(entity.key, self._kind, self._ancestor,
+  self._kind, str(uuid.uuid4()))
+
+datastore_helper.add_properties(entity, {"content": unicode(content)})
+return entity
+
+
+def 

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709899#comment-15709899
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1469

[BEAM-646] Move TransformHierarchy Maintenance into it

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This reduces the complexity of Pipeline.applyInternal by keeping the
responsiblities to passing a node into the Transform Hierarchy,
enforcing name uniqueness, and causing the runner to expand the
PTransform. This logic is moved to the appropriate application sites.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
transform_hierarchy_maintenance_internally

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1469


commit ae6ac672e582148b846c3910cc7247700b3403cb
Author: Thomas Groh 
Date:   2016-11-29T22:29:47Z

Move TransformHierarchy Maintenance into it

This reduces the complexity of Pipeline.applyInternal by keeping the
responsiblities to passing a node into the Transform Hierarchy,
enforcing name uniqueness, and causing the runner to expand the
PTransform. This logic is moved to the appropriate application sites.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1469: [BEAM-646] Move TransformHierarchy Mainte...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1469

[BEAM-646] Move TransformHierarchy Maintenance into it

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This reduces the complexity of Pipeline.applyInternal by keeping the
responsiblities to passing a node into the Transform Hierarchy,
enforcing name uniqueness, and causing the runner to expand the
PTransform. This logic is moved to the appropriate application sites.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
transform_hierarchy_maintenance_internally

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1469


commit ae6ac672e582148b846c3910cc7247700b3403cb
Author: Thomas Groh 
Date:   2016-11-29T22:29:47Z

Move TransformHierarchy Maintenance into it

This reduces the complexity of Pipeline.applyInternal by keeping the
responsiblities to passing a node into the Transform Hierarchy,
enforcing name uniqueness, and causing the runner to expand the
PTransform. This logic is moved to the appropriate application sites.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1468: [BEAM] Add table schema from json parser

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1468

[BEAM] Add table schema from json parser

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-add-table-schema-from-json-parser

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1468.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1468


commit 8620095eb2024a52427249386b412e318d676e44
Author: Sourabh Bajaj 
Date:   2016-11-30T21:48:28Z

Parse table schema from JSON




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1050) PipelineResult.State is not set to FAILED when a streaming job fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709693#comment-15709693
 ] 

ASF GitHub Bot commented on BEAM-1050:
--

Github user staslev closed the pull request at:

https://github.com/apache/incubator-beam/pull/1438


> PipelineResult.State is not set to FAILED when a streaming job fails
> 
>
> Key: BEAM-1050
> URL: https://issues.apache.org/jira/browse/BEAM-1050
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>Priority: Minor
>
> In case of failure, {{SteamingContext#awaitTerminationOrTimeout}} and 
> {{SteamingContext#awaitTermination}} will both throw an exception, and so 
> {{state = State.DONE}} will not be executed in the code block below. 
> In addition, it would probably make sense to set {{state = State.FAILED}} in 
> cases where an exception takes place.
> {code:java}
> if (isStreamingPipeline()) {
>   // stop streaming context
>   if (timeout > 0) {
> jssc.awaitTerminationOrTimeout(timeout);
>   } else {
> jssc.awaitTermination();
>   }
>   // stop streaming context gracefully, so checkpointing (and other 
> computations) get to
>   // finish before shutdown.
>   jssc.stop(false, gracefully);
> }
> state = State.DONE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1438: [BEAM-1050] Fixing pipeline's result stat...

[staslev]

Github user staslev closed the pull request at:

https://github.com/apache/incubator-beam/pull/1438


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1467: Improve size estimation speed for file sa...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1467

Improve size estimation speed for file samples

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-remove-full-size-reading-sample

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1467.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1467


commit f1cbcee35170c2f93e606a3da0c42e145085f308
Author: Sourabh Bajaj 
Date:   2016-11-30T20:18:04Z

Improve size estimation speed for file samples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1000) Add non blocking cancel() and waitUntilFinish() for Spark batch application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709611#comment-15709611
 ] 

ASF GitHub Bot commented on BEAM-1000:
--

GitHub user staslev opened a pull request:

https://github.com/apache/incubator-beam/pull/1466

Aligned the Spark batch and streaming pipelines to adhere to 
waitUntilFinish and cancel

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Introduced SparkPipelineResult that should be able to better support 
PipelineResult semantics, such as waitUntilFinish and cancel. 

Also relates to BEAM-1000,  BEAM-595 and BEAM-1050 as well as to #1438.

R: @amitsela 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/incubator-beam spark-pipeline-result

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1466.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1466


commit 63ba243c54dc50567fe73f88f2e8d17466121987
Author: Stas Levin 
Date:   2016-11-28T09:11:10Z

Added a SparkPipelineResult class to address 
PipelineResult#waitUntilFinish() and SparkRunner#run() semantics, also relates 
to BEAM-1000,  BEAM-595 and BEAM-1050.




> Add non blocking cancel() and waitUntilFinish() for Spark batch application
> ---
>
> Key: BEAM-1000
> URL: https://issues.apache.org/jira/browse/BEAM-1000
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Kobi Salant
>
> Spark batch submit by nature is blocking. 
> To enable cancel and waitForFinish we will need to run it in a different 
> thread/future and return a context which communicates with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1466: Aligned the Spark batch and streaming pip...

[staslev]

GitHub user staslev opened a pull request:

https://github.com/apache/incubator-beam/pull/1466

Aligned the Spark batch and streaming pipelines to adhere to 
waitUntilFinish and cancel

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Introduced SparkPipelineResult that should be able to better support 
PipelineResult semantics, such as waitUntilFinish and cancel. 

Also relates to BEAM-1000,  BEAM-595 and BEAM-1050 as well as to #1438.

R: @amitsela 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/incubator-beam spark-pipeline-result

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1466.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1466


commit 63ba243c54dc50567fe73f88f2e8d17466121987
Author: Stas Levin 
Date:   2016-11-28T09:11:10Z

Added a SparkPipelineResult class to address 
PipelineResult#waitUntilFinish() and SparkRunner#run() semantics, also relates 
to BEAM-1000,  BEAM-595 and BEAM-1050.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1421: Improve the GCS stat using batchAPI

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/incubator-beam/pull/1421


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1404: Improve the speed of getting file sizes u...

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/incubator-beam/pull/1404


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1465: Update examples archetype with runner pro...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1465

Update examples archetype with runner profiles

This makes it possible to run the examples on all runners.

R: @davorbonaci @bjchambers 

please be skeptical.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam starter-pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1465.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1465


commit 2a9b633f567cf955c46e3380b8ac174ccfedb415
Author: Dan Halperin 
Date:   2016-11-30T19:10:20Z

Update examples archetype with runner profiles

This makes it possible to run the examples on all runners.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify #1947

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1382: Support @ValidatesRunner(RunnableOnServic...

[markflyhigh]

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1382


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-747) Text checksum verifier is not resilient to eventually consistent filesystems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709256#comment-15709256
 ] 

ASF GitHub Bot commented on BEAM-747:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1189


> Text checksum verifier is not resilient to eventually consistent filesystems
> 
>
> Key: BEAM-747
> URL: https://issues.apache.org/jira/browse/BEAM-747
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> Example 1: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console
> Here it looks like we need to retry listing files, at least a little bit, if 
> none are found. They did show up:
> {code}
> gsutil ls 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
> {code}
> Example 2: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1525/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> Here it looks like we need to fill in the shard template if the filesystem 
> does not give us a consistent result:
> {code}
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [0 of 1] Read 162 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-0-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [1 of 1] Read 144 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-2-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> matchesSafely
> INFO: Generated checksum for output data: 
> aec68948b2515e6ea35fd1ed7649c267a10a01e5
> {code}
> We missed shard 1-of-3 and hence got the wrong checksum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1189: [BEAM-747] Improve FileChecksumMatcher Th...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-747] Fix FileChecksumMatcher That Inconsistent With FS

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8042d52fc -> b75a76459


[BEAM-747] Fix FileChecksumMatcher That Inconsistent With FS


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d08a9f12
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d08a9f12
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d08a9f12

Branch: refs/heads/master
Commit: d08a9f1278e0bb4ec5b08e11f6267c516c8ea56e
Parents: 8042d52
Author: Mark Liu 
Authored: Tue Oct 25 14:57:13 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Nov 30 09:52:34 2016 -0800

--
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 168 ---
 .../sdk/testing/FileChecksumMatcherTest.java| 131 ++-
 3 files changed, 268 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d08a9f12/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
index f2afe6a..01438de 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
@@ -65,7 +65,7 @@ public class WordCountIT {
 "output",
 "results"));
 options.setOnSuccessMatcher(
-new FileChecksumMatcher(DEFAULT_OUTPUT_CHECKSUM, options.getOutput() + 
"*"));
+new FileChecksumMatcher(DEFAULT_OUTPUT_CHECKSUM, options.getOutput() + 
"*-of-*"));
 
 WordCount.main(TestPipeline.convertToArgs(options));
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d08a9f12/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
index de6cea3..4b249fe 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
@@ -19,8 +19,14 @@
 package org.apache.beam.sdk.testing;
 
 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
 
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
 import com.google.common.hash.HashCode;
 import com.google.common.hash.Hashing;
 import com.google.common.io.CharStreams;
@@ -28,14 +34,21 @@ import java.io.IOException;
 import java.io.Reader;
 import java.nio.channels.Channels;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import javax.annotation.Nonnull;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.util.FluentBackoff;
 import org.apache.beam.sdk.util.IOChannelFactory;
 import org.apache.beam.sdk.util.IOChannelUtils;
 import org.hamcrest.Description;
 import org.hamcrest.TypeSafeMatcher;
+import org.joda.time.Duration;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -43,71 +56,172 @@ import org.slf4j.LoggerFactory;
  * Matcher to verify file checksum in E2E test.
  *
  * For example:
- * {@code [
- *   assertTrue(job, new FileChecksumMatcher(checksumString, filePath));
- * ]}
+ * {@code
+ *   assertThat(job, new FileChecksumMatcher(checksumString, filePath));
+ * }
+ * or
+ * {@code
+ *   assertThat(job, new FileChecksumMatcher(checksumString, filePath, 
shardTemplate));
+ * }
+ *
+ * Checksum of outputs is generated based on SHA-1 algorithm. If output 
file is empty,
+ * SHA-1 hash of empty string (da39a3ee5e6b4b0d3255bfef95601890afd80709) is 
used as expected.
  */
 public class FileChecksumMatcher extends TypeSafeMatcher
 implements SerializableMatcher {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(FileChecksumMatcher.class);
 
+  static final int MAX_READ_RETRIES = 4;
+  static final Duration DEFAULT_SLEEP_DURATION = Duration.standardSeconds(10L);
+  static final FluentBackoff BACK_OFF_FACTORY =
+  FluentBackoff.DEFAULT

[2/2] incubator-beam git commit: This closes #1189

[kenn]

This closes #1189


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b75a7645
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b75a7645
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b75a7645

Branch: refs/heads/master
Commit: b75a76459e18c032af7981197acd0c1fb826bc97
Parents: 8042d52 d08a9f1
Author: Kenneth Knowles 
Authored: Wed Nov 30 09:53:02 2016 -0800
Committer: Kenneth Knowles 
Committed: Wed Nov 30 09:53:02 2016 -0800

--
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 168 ---
 .../sdk/testing/FileChecksumMatcherTest.java| 131 ++-
 3 files changed, 268 insertions(+), 33 deletions(-)
--

[jira] [Commented] (BEAM-607) Add DistributedLog IO

[Khurrum Nasim (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708365#comment-15708365
 ] 

Khurrum Nasim commented on BEAM-607:


[~jbonofre] I just sent out the first pull request for adding a bounded source 
for DistributedLog streams. It is in 
https://github.com/apache/incubator-beam/pull/1464 Let me know if you have any 
questions. I will send out an unbounded source and sink soon.

> Add DistributedLog IO
> -
>
> Key: BEAM-607
> URL: https://issues.apache.org/jira/browse/BEAM-607
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Khurrum Nasim
>
> I'd like to add an IO for the new DistributedLog streams - 
> http://distributedlog.io
> - bounded source and sink (sealed streams)
> - unbounded source and sink (unsealed streams)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1464: BEAM-607: Add a bounded source for Apache...

[khurrumnasimm]

GitHub user khurrumnasimm opened a pull request:

https://github.com/apache/incubator-beam/pull/1464

BEAM-607: Add a bounded source for Apache DistributedLog (incubating)

Apache DistributedLog (incubating) is a replicated log stream store. This 
pull request is the first part of BEAM-607 - ADD a bounded source for Apache 
DistributedLog streams.

The bounded source is similar as HDFS bounded source.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/khurrumnasimm/incubator-beam 
kn/distributedlog_io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1464


commit f0db56a7eb3ac840873cc9e3acf7a997d383f583
Author: Khurrum Nasim 
Date:   2016-11-30T11:41:42Z

Add a bounded source for Apache DistributedLog (incubating)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-607) Add DistributedLog IO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708357#comment-15708357
 ] 

ASF GitHub Bot commented on BEAM-607:
-

GitHub user khurrumnasimm opened a pull request:

https://github.com/apache/incubator-beam/pull/1464

BEAM-607: Add a bounded source for Apache DistributedLog (incubating)

Apache DistributedLog (incubating) is a replicated log stream store. This 
pull request is the first part of BEAM-607 - ADD a bounded source for Apache 
DistributedLog streams.

The bounded source is similar as HDFS bounded source.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/khurrumnasimm/incubator-beam 
kn/distributedlog_io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1464


commit f0db56a7eb3ac840873cc9e3acf7a997d383f583
Author: Khurrum Nasim 
Date:   2016-11-30T11:41:42Z

Add a bounded source for Apache DistributedLog (incubating)




> Add DistributedLog IO
> -
>
> Key: BEAM-607
> URL: https://issues.apache.org/jira/browse/BEAM-607
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Khurrum Nasim
>
> I'd like to add an IO for the new DistributedLog streams - 
> http://distributedlog.io
> - bounded source and sink (sealed streams)
> - unbounded source and sink (unsealed streams)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-932) Findbugs doesn't pass in Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708090#comment-15708090
 ] 

ASF GitHub Bot commented on BEAM-932:
-

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam/pull/1463

[BEAM-932] Enable findbugs validation (and fix existing issues)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam BEAM-932-findbugs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1463


commit 97a1c00a8c4be54af73c44fef4aaad988168b0f4
Author: Ismaël Mejía 
Date:   2016-11-26T13:24:37Z

[BEAM-932] Enable findbugs validation (and fix existing issues)




> Findbugs doesn't pass in Spark runner
> -
>
> Key: BEAM-932
> URL: https://issues.apache.org/jira/browse/BEAM-932
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-runners-spark 
> ---
> [INFO] BugInstance size is 19
> [INFO] Error size is 0
> [INFO] Total bugs: 19
> [INFO] instanceof will always return false in 
> org.apache.beam.runners.spark.SparkRunner.run(Pipeline), since a 
> RuntimeException can't be a org.apache.spark.SparkException 
> [org.apache.beam.runners.spark.SparkRunner] At SparkRunner.java:[line 161]
> [INFO] 
> org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1.apply(Map$Entry)
>  makes inefficient use of keySet iterator instead of entrySet iterator 
> [org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1]
>  At WithNamedAggregatorsSupport.java:[line 125]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink shadows the 
> simple name of the superclass org.apache.spark.metrics.sink.CsvSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink] At 
> CsvSink.java:[lines 37-38]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink shadows 
> the simple name of the superclass org.apache.spark.metrics.sink.GraphiteSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink] At 
> GraphiteSink.java:[lines 37-38]
> [INFO] t must be non-null but is marked as nullable 
> [org.apache.beam.runners.spark.translation.WindowingHelpers$4] At 
> WindowingHelpers.java:[line 88]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1.evaluate(ConsoleIO$Write$Unbound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1]
>  At StreamingTransformTranslator.java:[line 92]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10.evaluate(ParDo$Bound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10]
>  At StreamingTransformTranslator.java:[line 360]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11.evaluate(ParDo$BoundMulti,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11]
>  At StreamingTransformTranslator.java:[line 395]
> [INFO] Unchecked/unconfirmed cast from 
> 

[GitHub] incubator-beam pull request #1463: [BEAM-932] Enable findbugs validation (an...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam/pull/1463

[BEAM-932] Enable findbugs validation (and fix existing issues)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam BEAM-932-findbugs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1463


commit 97a1c00a8c4be54af73c44fef4aaad988168b0f4
Author: Ismaël Mejía 
Date:   2016-11-26T13:24:37Z

[BEAM-932] Enable findbugs validation (and fix existing issues)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---