Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #149

[Apache Jenkins Server]

See 


Changes:

[owenzhang1990] [BEAM-79] fix integration-test failure

--
[...truncated 11175 lines...]
at 
org.apache.beam.runners.gearpump.GearpumpPipelineTranslator.translate(GearpumpPipelineTranslator.java:82)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.run(GearpumpPipelineRunner.java:111)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.run(TestGearpumpRunner.java:54)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.run(TestGearpumpRunner.java:34)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:113)
at 
org.apache.beam.sdk.transforms.ApproximateUniqueTest.testApproximateUniqueWithSmallInput(ApproximateUniqueTest.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Running org.apache.beam.sdk.transforms.windowing.WindowingTest
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.584 sec <<< 
FAILURE! - in org.apache.beam.sdk.transforms.windowing.WindowingTest
testMergingWindowing(org.apache.beam.sdk.transforms.windowing.WindowingTest)  
Time elapsed: 0.637 sec  <<< ERROR!
java.lang.UnsupportedOperationException: merging window is not supported in 
Gearpump pipeline
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner$AssignWindowsAndSetStrategy.apply(GearpumpPipelineRunner.java:167)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner$AssignWindowsAndSetStrategy.apply(GearpumpPipelineRunner.java:149)
at 
org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:77)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.apply(GearpumpPipelineRunner.java:81)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.apply(TestGearpumpRunner.java:62)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:323)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:167)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest$WindowedCount.apply(WindowingTest.java:74)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest$WindowedCount.apply(WindowingTest.java:56)
at 
org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:77)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.apply(GearpumpPipelineRunner.java:93)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.apply(TestGearpumpRunner.java:62)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest.testMergingWindowing(WindowingTest.java:153)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489387#comment-15489387
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/956


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-598) GearpumpRunner depends on SNAPSHOT jar that has since been released and deleted

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-598.

   Resolution: Fixed
Fix Version/s: Not applicable

> GearpumpRunner depends on SNAPSHOT jar that has since been released and 
> deleted
> ---
>
> Key: BEAM-598
> URL: https://issues.apache.org/jira/browse/BEAM-598
> Project: Beam
>  Issue Type: Bug
>  Components: runner-gearpump
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Manu Zhang
> Fix For: Not applicable
>
>
> {code}
> [ERROR] Failed to execute goal on project beam-runners-gearpump: Could not 
> resolve dependencies for project 
> org.apache.beam:beam-runners-gearpump:jar:0.3.0-incubating-SNAPSHOT: The 
> following artifacts could not be resolved: 
> org.apache.gearpump:gearpump-streaming_2.11:jar:0.8.1-SNAPSHOT, 
> org.apache.gearpump:gearpump-core_2.11:jar:0.8.1-SNAPSHOT, 
> org.apache.gearpump:gearpump-daemon_2.11:jar:0.8.1-SNAPSHOT: Failure to find 
> org.apache.gearpump:gearpump-streaming_2.11:jar:0.8.1-SNAPSHOT in 
> http://repository.apache.org/snapshots was cached in the local repository, 
> resolution will not be reattempted until the update interval of Nexus has 
> elapsed or updates are forced -> [Help 1]
> {code}
> https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/145/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-79] fix integration-test failure

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/gearpump-runner ed7c4aaf9 -> 8f4334cf8


[BEAM-79] fix integration-test failure


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/272fe9f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/272fe9f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/272fe9f6

Branch: refs/heads/gearpump-runner
Commit: 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
Parents: ed7c4aa
Author: manuzhang 
Authored: Wed Sep 14 11:20:27 2016 +0800
Committer: manuzhang 
Committed: Wed Sep 14 11:28:05 2016 +0800

--
 runners/gearpump/pom.xml | 55 +--
 1 file changed, 27 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/272fe9f6/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 296de6b..6576ba6 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -32,6 +32,14 @@
   Apache Beam :: Runners :: Gearpump
   jar
 
+  
+
+  apache-repo
+  apache maven repo
+  https://repository.apache.org/content/repositories/releases
+
+  
+
   
 UTF-8
 UTF-8
@@ -47,22 +55,24 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-2.19.1
 
   
 runnable-on-service-tests
+integration-test
+
+  test
+
 
   
org.apache.beam.sdk.testing.RunnableOnService
   none
   true
   
 
org.apache.beam:beam-sdks-java-core
-
org.apache.beam:beam-runners-java-core
   
   
 
 
-  org.apache.beam.sdk.io.BigQueryIOTest,
+  org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest,
   org.apache.beam.sdk.io.CountingInputTest,
   org.apache.beam.sdk.io.CountingSourceTest,
   org.apache.beam.sdk.testing.PAssertTest,
@@ -189,6 +199,20 @@
   guava
 
 
+  org.apache.gearpump
+  gearpump-shaded-metrics-graphite_2.11
+  ${gearpump.version}
+  assembly
+  test
+
+
+  org.apache.gearpump
+  gearpump-shaded-guava_2.11
+  ${gearpump.version}
+  assembly
+  test
+
+
   junit
   junit
   test
@@ -218,9 +242,7 @@
 
   com.google.auto.service
   auto-service
-  1.0-rc2
 
-
   
 
   
@@ -245,29 +267,6 @@
 maven-compiler-plugin
   
 
-  
-  
-maven-failsafe-plugin
-
-  
-
-  integration-test
-  verify
-
-  
-
-
-  1
-  -Dlog4j.configuration=log4j-test.properties  
-XX:-UseGCOverheadLimit
-
-  
-
-  
-  
-org.apache.maven.plugins
-maven-surefire-plugin
-  
-
   
   
 org.apache.maven.plugins

[2/2] incubator-beam git commit: Closes #956

[dhalperi]

Closes #956


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8f4334cf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8f4334cf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8f4334cf

Branch: refs/heads/gearpump-runner
Commit: 8f4334cf8ac7f27005c730a9d7217d2afb0f6245
Parents: ed7c4aa 272fe9f
Author: Dan Halperin 
Authored: Tue Sep 13 21:32:35 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 13 21:32:35 2016 -0700

--
 runners/gearpump/pom.xml | 55 +--
 1 file changed, 27 insertions(+), 28 deletions(-)
--

[GitHub] incubator-beam pull request #956: [BEAM-79] fix integration-test failure

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/956


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #957: Remove PubsubFileInjector and IntraBundleP...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/957

Remove PubsubFileInjector and IntraBundleParallelization





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam archetypes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/957.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #957


commit fc96474804cdcd950663e7eaea5c140652a07c8e
Author: peihe 
Date:   2016-09-14T04:29:13Z

Remove PubsubFileInjector and IntraBundleParallelization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489256#comment-15489256
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/956

[BEAM-79] fix integration-test failure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #956


commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
Author: manuzhang 
Date:   2016-09-14T03:20:27Z

[BEAM-79] fix integration-test failure




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #956: [BEAM-79] fix integration-test failure

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/956

[BEAM-79] fix integration-test failure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #956


commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
Author: manuzhang 
Date:   2016-09-14T03:20:27Z

[BEAM-79] fix integration-test failure




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1141

[Apache Jenkins Server]

See 

[3/6] incubator-beam git commit: Additinal examples code cleanups

[lcwik]

Additinal examples code cleanups


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7fbe4103
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7fbe4103
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7fbe4103

Branch: refs/heads/master
Commit: 7fbe41035b775b631d5e9120d005497813d9e36c
Parents: b9a66e4
Author: Pei He 
Authored: Fri Sep 9 14:15:49 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 18:01:08 2016 -0700

--
 .../main/java/org/apache/beam/examples/MinimalWordCount.java| 3 +--
 .../main/java/org/apache/beam/examples/common/ExampleUtils.java | 2 +-
 .../archetype-resources/src/main/java/DebuggingWordCount.java   | 1 -
 .../archetype-resources/src/main/java/MinimalWordCount.java | 3 +--
 .../src/main/java/common/ExampleOptions.java| 5 -
 .../archetype-resources/src/main/java/common/ExampleUtils.java  | 2 +-
 6 files changed, 4 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7fbe4103/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
index f772dd5..14ffa18 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
@@ -75,8 +75,7 @@ public class MinimalWordCount {
 //   
dataflowOptions.setTempLocation("gs://SET_YOUR_BUCKET_NAME_HERE/AND_TEMP_DIRECTORY");
 // For FlinkRunner, set the runner as follows. See {@code 
FlinkPipelineOptions}
 // for more details.
-//   options.as(FlinkPipelineOptions.class)
-//  .setRunner(FlinkRunner.class);
+//   options.setRunner(FlinkRunner.class);
 
 // Create the Pipeline object with the options we defined above.
 Pipeline p = Pipeline.create(options);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7fbe4103/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
index 2e8dcf6..1209a67 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
@@ -53,7 +53,7 @@ import org.joda.time.Duration;
  * The utility class that sets up and tears down external resources,
  * and cancels the streaming pipelines once the program terminates.
  *
- * It is used to run Beam examples, such as TrafficMaxLaneFlow and 
TrafficRoutes.
+ * It is used to run Beam examples.
  */
 public class ExampleUtils {
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7fbe4103/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
index e315ba9..4099a37 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
@@ -17,7 +17,6 @@
  */
 package ${package};
 
-import ${package}.WordCount;
 import java.util.Arrays;
 import java.util.List;
 import java.util.regex.Pattern;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7fbe4103/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/MinimalWordCount.java
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/MinimalWordCount.java
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/MinimalWordCount.java
index f739fd8..e8497c0 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/MinimalWordCount.java
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/MinimalWordCount.java
@@ -75,8 +75,7 @@ public class MinimalWordCount {
 //   

[6/6] incubator-beam git commit: Update Beam examples archetypes

[lcwik]

Update Beam examples archetypes

This closes #936


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e9a08e45
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e9a08e45
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e9a08e45

Branch: refs/heads/master
Commit: e9a08e454b20c88c72ea4ce3138df77f898b26d7
Parents: e3768f6 76f0ff4
Author: Luke Cwik 
Authored: Tue Sep 13 18:01:42 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 18:01:42 2016 -0700

--
 .../apache/beam/examples/MinimalWordCount.java  |   3 +-
 .../beam/examples/common/ExampleUtils.java  |   2 +-
 pom.xml |  14 +-
 sdks/java/maven-archetypes/examples/pom.xml |  25 ++
 .../main/resources/archetype-resources/pom.xml  |  20 +-
 .../src/main/java/DebuggingWordCount.java   |  32 +-
 .../src/main/java/MinimalWordCount.java |  50 ++-
 .../src/main/java/WindowedWordCount.java| 139 +++
 .../src/main/java/WordCount.java|  77 ++--
 .../java/common/DataflowExampleOptions.java |  32 --
 .../main/java/common/DataflowExampleUtils.java  | 391 ---
 .../common/ExampleBigQueryTableOptions.java |  11 +-
 .../src/main/java/common/ExampleOptions.java|  32 ++
 ...xamplePubsubTopicAndSubscriptionOptions.java |  45 +++
 .../java/common/ExamplePubsubTopicOptions.java  |  17 +-
 .../src/main/java/common/ExampleUtils.java  | 353 +
 .../main/java/common/PubsubFileInjector.java| 153 
 .../src/test/java/WordCountTest.java|   9 +-
 sdks/java/maven-archetypes/starter/pom.xml  |   8 +
 .../main/resources/archetype-resources/pom.xml  |   6 +-
 .../resources/projects/basic/reference/pom.xml  |   6 +-
 21 files changed, 641 insertions(+), 784 deletions(-)
--

[2/6] incubator-beam git commit: Update Beam examples archetypes

[lcwik]

Update Beam examples archetypes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b9a66e4b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b9a66e4b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b9a66e4b

Branch: refs/heads/master
Commit: b9a66e4b50ae1fe5fa3afc33b2523e2f9d64b2c4
Parents: e3768f6
Author: Pei He 
Authored: Thu Sep 8 19:16:12 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 18:01:08 2016 -0700

--
 .../src/main/java/DebuggingWordCount.java   |  31 +-
 .../src/main/java/MinimalWordCount.java |  51 ++-
 .../src/main/java/WindowedWordCount.java| 139 +++
 .../src/main/java/WordCount.java|  77 ++--
 .../java/common/DataflowExampleOptions.java |  32 --
 .../main/java/common/DataflowExampleUtils.java  | 391 ---
 .../common/ExampleBigQueryTableOptions.java |  11 +-
 .../src/main/java/common/ExampleOptions.java|  37 ++
 ...xamplePubsubTopicAndSubscriptionOptions.java |  45 +++
 .../java/common/ExamplePubsubTopicOptions.java  |  17 +-
 .../src/main/java/common/ExampleUtils.java  | 353 +
 .../main/java/common/PubsubFileInjector.java|   8 +-
 .../src/test/java/WordCountTest.java|   9 +-
 13 files changed, 592 insertions(+), 609 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b9a66e4b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
index e9f4333..e315ba9 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
@@ -28,7 +28,7 @@ import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Aggregator;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.values.KV;
@@ -36,8 +36,9 @@ import org.apache.beam.sdk.values.PCollection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+
 /**
- * An example that verifies word counts in Shakespeare and includes Dataflow 
best practices.
+ * An example that verifies word counts in Shakespeare and includes Beam best 
practices.
  *
  * This class, {@link DebuggingWordCount}, is the third in a series of four 
successively more
  * detailed 'word count' examples. You may first want to take a look at {@link 
MinimalWordCount}
@@ -46,12 +47,12 @@ import org.slf4j.LoggerFactory;
  *
  * Basic concepts, also in the MinimalWordCount and WordCount examples:
  * Reading text files; counting a PCollection; executing a Pipeline both 
locally
- * and using the Dataflow service; defining DoFns.
+ * and using a selected runner; defining DoFns.
  *
  * New Concepts:
  * 
  *   1. Logging to Cloud Logging
- *   2. Controlling Dataflow worker log levels
+ *   2. Controlling worker log levels
  *   3. Creating a custom aggregator
  *   4. Testing your Pipeline via PAssert
  * 
@@ -62,12 +63,14 @@ import org.slf4j.LoggerFactory;
  * }
  * 
  *
- * To execute this pipeline using the Dataflow service and the additional 
logging discussed
- * below, specify pipeline configuration:
+ * To change the runner, specify:
+ * {@code
+ *   --runner=YOUR_SELECTED_RUNNER
+ * }
+ * 
+ *
+ * To use the additional logging discussed below, specify:
  * {@code
- *   --project=YOUR_PROJECT_ID
- *   --stagingLocation=gs://YOUR_STAGING_DIRECTORY
- *   --runner=BlockingDataflowRunner
  *   --workerLogLevelOverrides={"org.apache.beam.examples":"DEBUG"}
  * }
  * 
@@ -100,12 +103,12 @@ import org.slf4j.LoggerFactory;
  * that changing the default worker log level to TRACE or DEBUG will 
significantly increase
  * the amount of logs output.
  *
- * The input file defaults to {@code 
gs://dataflow-samples/shakespeare/kinglear.txt} and can be
- * overridden with {@code --inputFile}.
+ * The input file defaults to {@code 
gs://apache-beam-samples/shakespeare/kinglear.txt}
+ * and can be overridden with {@code --inputFile}.
  */
 public class DebuggingWordCount {
   /** A DoFn that filters for a specific key based upon a regular 

[1/6] incubator-beam git commit: Remove PubsubFileInjector

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e3768f6da -> e9a08e454


Remove PubsubFileInjector


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1047e033
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1047e033
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1047e033

Branch: refs/heads/master
Commit: 1047e03383b535124b36ec09f900f4e8e3df3a23
Parents: 7fbe410
Author: Pei He 
Authored: Mon Sep 12 16:29:47 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 18:01:08 2016 -0700

--
 .../main/java/common/PubsubFileInjector.java| 153 ---
 1 file changed, 153 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1047e033/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/common/PubsubFileInjector.java
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/common/PubsubFileInjector.java
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/common/PubsubFileInjector.java
deleted file mode 100644
index 6ca20f3..000
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/common/PubsubFileInjector.java
+++ /dev/null
@@ -1,153 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package ${package}.common;
-
-import com.google.api.services.pubsub.Pubsub;
-import com.google.api.services.pubsub.model.PublishRequest;
-import com.google.api.services.pubsub.model.PubsubMessage;
-import com.google.common.collect.ImmutableMap;
-import java.io.IOException;
-import java.util.Arrays;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.options.PubsubOptions;
-import org.apache.beam.sdk.options.Validation;
-import org.apache.beam.sdk.transforms.IntraBundleParallelization;
-import org.apache.beam.sdk.transforms.OldDoFn;
-import org.apache.beam.sdk.util.Transport;
-
-/**
- * A batch Dataflow pipeline for injecting a set of GCS files into
- * a PubSub topic line by line. Empty lines are skipped.
- *
- * This is useful for testing streaming
- * pipelines. Note that since batch pipelines might retry chunks, this
- * does _not_ guarantee exactly-once injection of file data. Some lines may
- * be published multiple times.
- * 
- */
-public class PubsubFileInjector {
-
-  /**
-   * An incomplete {@code PubsubFileInjector} transform with unbound output 
topic.
-   */
-  public static class Unbound {
-private final String timestampLabelKey;
-
-Unbound() {
-  this.timestampLabelKey = null;
-}
-
-Unbound(String timestampLabelKey) {
-  this.timestampLabelKey = timestampLabelKey;
-}
-
-Unbound withTimestampLabelKey(String timestampLabelKey) {
-  return new Unbound(timestampLabelKey);
-}
-
-public Bound publish(String outputTopic) {
-  return new Bound(outputTopic, timestampLabelKey);
-}
-  }
-
-  /** A {@link OldDoFn} that publishes non-empty lines to Google Cloud PubSub. 
*/
-  public static class Bound extends OldDoFn {
-private final String outputTopic;
-private final String timestampLabelKey;
-public transient Pubsub pubsub;
-
-public Bound(String outputTopic, String timestampLabelKey) {
-  this.outputTopic = outputTopic;
-  this.timestampLabelKey = timestampLabelKey;
-}
-
-@Override
-public void startBundle(Context context) {
-  this.pubsub =
-  
Transport.newPubsubClient(context.getPipelineOptions().as(PubsubOptions.class))
-  .build();
-}
-
-@Override
-public void processElement(ProcessContext c) throws IOException {
-  if 

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488969#comment-15488969
 ] 

Eugene Kirpichov commented on BEAM-630:
---

Hmm, looks like I don't have the rights to close this issue. Dan, can you 
either close it or grant me the rights? or am I missing something?

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-493) All Runners Run WordCount in Precommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488932#comment-15488932
 ] 

ASF GitHub Bot commented on BEAM-493:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/931


> All Runners Run WordCount in Precommit
> --
>
> Key: BEAM-493
> URL: https://issues.apache.org/jira/browse/BEAM-493
> Project: Beam
>  Issue Type: Improvement
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-493] Modify examples pom to add precommit execution profile.

[lcwik]

[BEAM-493] Modify examples pom to add precommit execution profile.

This closes #931


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e3768f6d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e3768f6d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e3768f6d

Branch: refs/heads/master
Commit: e3768f6da9786ba60317fcd74e0fa06c26b01acc
Parents: 5979b61 1c96e59
Author: Luke Cwik 
Authored: Tue Sep 13 17:15:46 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 17:15:46 2016 -0700

--
 examples/java/pom.xml | 103 +
 1 file changed, 103 insertions(+)
--

[1/2] incubator-beam git commit: Modify examples pom to add precommit execution profile.

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5979b61b9 -> e3768f6da


Modify examples pom to add precommit execution profile.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1c96e590
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1c96e590
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1c96e590

Branch: refs/heads/master
Commit: 1c96e5901c520632d1682b660e0e791cfcf966ad
Parents: 5979b61
Author: Jason Kuster 
Authored: Wed Sep 7 17:12:11 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 17:15:21 2016 -0700

--
 examples/java/pom.xml | 103 +
 1 file changed, 103 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1c96e590/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 47f0366..6a39f64 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -114,6 +114,108 @@
 
   
 
+
+  jenkins-precommit
+  
+WordCountIT
+false
+true
+  
+  
+
+  
+org.apache.maven.plugins
+maven-failsafe-plugin
+
+  false
+  true
+
+
+  
+direct-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempRoot=gs://temp-storage-for-end-to-end-tests",
+  "--runner=org.apache.beam.runners.direct.DirectRunner"
+  ]
+
+  
+
+  
+  
+flink-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempRoot=gs://temp-storage-for-end-to-end-tests",
+  "--runner=org.apache.beam.runners.flink.TestFlinkRunner"
+  ]
+
+  
+
+  
+  
+spark-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempRoot=/tmp",
+  "--runner=org.apache.beam.runners.spark.TestSparkRunner"
+  ]
+
+  
+
+  
+  
+dataflow-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempRoot=gs://temp-storage-for-end-to-end-tests",
+  
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner"
+  ]
+
+  
+
+  
+
+  
+
+  
+
   
 
   
@@ -269,6 +371,7 @@
   verify
 
 
+  ${skipDefaultIT}
   all
   4
   

[GitHub] incubator-beam pull request #955: Implement dynamic work rebalancing for con...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/955

Implement dynamic work rebalancing for concat sources.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam concat

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/955.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #955


commit 0c359031b589d12b4ec6a32669aedf9409a74752
Author: Robert Bradshaw 
Date:   2016-09-13T23:25:33Z

Minor cleanups in docstrings and error messages.

commit 3f7fb80b24f87cd10b0eb7fd6a2a7d081f5903f8
Author: Robert Bradshaw 
Date:   2016-09-13T23:26:48Z

Fix bug in offset range tracker splitting.

The fraction must be computed before resetting the endpoint,
otherwise it will always be 1.

commit 7f788b4414865182ff6adacbea19230e1a0c812c
Author: Robert Bradshaw 
Date:   2016-09-13T23:37:06Z

Implement liquid sharding for concat source.

commit 9a4054241178cb967ee309e4e8ece2e3f0fe1c41
Author: Robert Bradshaw 
Date:   2016-09-13T23:49:22Z

Move ConcatSource to iobase.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488859#comment-15488859
 ] 

ASF GitHub Bot commented on BEAM-630:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/953


> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-630] Fixes wrong type name in generated code

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0800a8cb2 -> 5979b61b9


[BEAM-630] Fixes wrong type name in generated code


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2f825177
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2f825177
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2f825177

Branch: refs/heads/master
Commit: 2f825177c59add53fa259325c2fe17981a707ba1
Parents: 0800a8c
Author: Eugene Kirpichov 
Authored: Tue Sep 13 14:19:29 2016 -0700
Committer: Eugene Kirpichov 
Committed: Tue Sep 13 14:19:29 2016 -0700

--
 .../java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2f825177/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
index f622015..edc1dc0 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
@@ -397,7 +397,7 @@ public class DoFnInvokers {
 private Object describeType(Type type) {
   switch (type.getSort()) {
 case Type.OBJECT:
-  return type.getDescriptor();
+  return type.getInternalName();
 case Type.INT:
 case Type.BYTE:
 case Type.BOOLEAN:

[2/2] incubator-beam git commit: Closes #953

[dhalperi]

Closes #953


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5979b61b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5979b61b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5979b61b

Branch: refs/heads/master
Commit: 5979b61b9a8373fd33daf5226352ae2802fd92d7
Parents: 0800a8c 2f82517
Author: Dan Halperin 
Authored: Tue Sep 13 16:53:05 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 13 16:53:05 2016 -0700

--
 .../java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #953: [BEAM-630] Fixes wrong type name in genera...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/953


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #948: Simple streaming text sink with windowing

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/948


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488494#comment-15488494
 ] 

ASF GitHub Bot commented on BEAM-630:
-

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/954

Enables BytecodeVerificationLocal on travis

This is to achieve stricter verification of bytecode,
mainly bytecode generated by DoFnInvokers dynamically.

See https://issues.apache.org/jira/browse/BEAM-630 for
an example situation that could have been caught earlier
with this change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bytecode-verify

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/954.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #954






> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #954: Enables BytecodeVerificationLocal on travi...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/954

Enables BytecodeVerificationLocal on travis

This is to achieve stricter verification of bytecode,
mainly bytecode generated by DoFnInvokers dynamically.

See https://issues.apache.org/jira/browse/BEAM-630 for
an example situation that could have been caught earlier
with this change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bytecode-verify

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/954.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #954






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488439#comment-15488439
 ] 

Eugene Kirpichov commented on BEAM-630:
---

WELL, ACTUALLY it might be our bug. Almost narrowed down.

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488470#comment-15488470
 ] 

ASF GitHub Bot commented on BEAM-630:
-

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/953

[BEAM-630] Fixes wrong type name in generated code

See https://issues.apache.org/jira/browse/BEAM-630 for details. We 
generated the wrong local variable type (with a semicolon - "descriptor" - 
rather than without a semicolon - "internal name"). It only failed when JVM was 
using sufficiently strict settings when verifying the bytecode.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam fix-invokers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #953


commit 2f825177c59add53fa259325c2fe17981a707ba1
Author: Eugene Kirpichov 
Date:   2016-09-13T21:19:29Z

[BEAM-630] Fixes wrong type name in generated code




> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #953: [BEAM-630] Fixes wrong type name in genera...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/953

[BEAM-630] Fixes wrong type name in generated code

See https://issues.apache.org/jira/browse/BEAM-630 for details. We 
generated the wrong local variable type (with a semicolon - "descriptor" - 
rather than without a semicolon - "internal name"). It only failed when JVM was 
using sufficiently strict settings when verifying the bytecode.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam fix-invokers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #953


commit 2f825177c59add53fa259325c2fe17981a707ba1
Author: Eugene Kirpichov 
Date:   2016-09-13T21:19:29Z

[BEAM-630] Fixes wrong type name in generated code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488397#comment-15488397
 ] 

Eugene Kirpichov edited comment on BEAM-630 at 9/13/16 9:01 PM:


Wonderful. This seems like a bug in ByteBuddy - it legitimately produces a 
broken class file (the class name that the JVM complains is illegal by that 
link, "Lorg/apache/beam/sdk/transforms/DoFn$ProcessContext;", is actually 
illegal cause it contains a semicolon - see 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.1, 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.2), but 
in some cases JVM skips verifying it. With -DforkCount=0 it doesn't, for some 
reason I don't quite understand yet (and it probably doesn't matter), but with 
-XX:+BytecodeVerificationLocal it fails even when run as a test method from 
IDEA.

Don't ask how I found out (hint: I found out by reading JVM source code).

To answer the implicit question "why does anything work". Seems like ByteBuddy 
generated a constant pool entry of type "class" containing this invalid class 
name, but the entry is not used in the rest of the code. So it fails 
verification, but doesn't cause anything to break at runtime. Weird.


was (Author: jkff):
Wonderful. This seems like a bug in ByteBuddy - it legitimately produces a 
broken class file (the class name that the JVM complains is illegal by that 
link, "Lorg/apache/beam/sdk/transforms/DoFn$ProcessContext;", is actually 
illegal cause it contains a semicolon - see 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.1, 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.2), but 
in some cases JVM skips verifying it. With -DforkCount=0 it doesn't, for some 
reason I don't quite understand yet (and it probably doesn't matter), but with 
-XX:+BytecodeVerificationLocal it fails even when run as a test method from 
IDEA.

Don't ask how I found out (hint: I found out by reading JVM source code).

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488397#comment-15488397
 ] 

Eugene Kirpichov edited comment on BEAM-630 at 9/13/16 8:57 PM:


Wonderful. This seems like a bug in ByteBuddy - it legitimately produces a 
broken class file (the class name that the JVM complains is illegal by that 
link, "Lorg/apache/beam/sdk/transforms/DoFn$ProcessContext;", is actually 
illegal cause it contains a semicolon - see 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.1, 
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.2.2), but 
in some cases JVM skips verifying it. With -DforkCount=0 it doesn't, for some 
reason I don't quite understand yet (and it probably doesn't matter), but with 
-XX:+BytecodeVerificationLocal it fails even when run as a test method from 
IDEA.

Don't ask how I found out (hint: I found out by reading JVM source code).


was (Author: jkff):
Wonderful. This seems like a bug in ByteBuddy - it legitimately produces a 
broken class file, but in some cases JVM skips verifying it. With -DforkCount=0 
it doesn't, for some reason I don't quite understand yet (and it probably 
doesn't matter), but with -XX:+BytecodeVerificationLocal it fails even when run 
as a test method from IDEA.

Don't ask how I found out (hint: I found out by reading JVM source code).

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488397#comment-15488397
 ] 

Eugene Kirpichov commented on BEAM-630:
---

Wonderful. This seems like a bug in ByteBuddy - it legitimately produces a 
broken class file, but in some cases JVM skips verifying it. With -DforkCount=0 
it doesn't, for some reason I don't quite understand yet (and it probably 
doesn't matter), but with -XX:+BytecodeVerificationLocal it fails even when run 
as a test method from IDEA.

Don't ask how I found out (hint: I found out by reading JVM source code).

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-630) DataflowRunnableOnService test suite is broken

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488356#comment-15488356
 ] 

Eugene Kirpichov commented on BEAM-630:
---

Some investigation:
- The issue can be narrowed down to "-DforkCount=0" - "mvn clean test 
-DforkCount=0" fails in this way, but a simple "mvn clean test" passes.
- The class files generated with and without this option are identical.
- On my machine, system properties with and without this option are nearly 
identical: the differing properties are classworlds.conf, java.class.path, 
magen.home, maven.multiModuleProjectDirectory, securerandom.source, 
sun.java.command. The JVM version et al. are the same. I don't see how any of 
these properties could affect whether or not the class verifies.

> DataflowRunnableOnService test suite is broken
> --
>
> Key: BEAM-630
> URL: https://issues.apache.org/jira/browse/BEAM-630
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>
> Looks like it's related to problems with the new DoFnSignatures/Invoker: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-436) DirectRunner RunnableOnService tempLocation configuration insufficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488207#comment-15488207
 ] 

ASF GitHub Bot commented on BEAM-436:
-

Github user swegner closed the pull request at:

https://github.com/apache/incubator-beam/pull/753


> DirectRunner RunnableOnService tempLocation configuration insufficient
> --
>
> Key: BEAM-436
> URL: https://issues.apache.org/jira/browse/BEAM-436
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Scott Wegner
>Assignee: Davor Bonaci
>
> The way we specify temp storage locations for RunnableOnService tests is not 
> sufficient, specifically for DirectRunner execution. Right now, 
> RunnableOnService tests are run for DirectRunner and DataflowRunner, who set 
> their temp locations differently:
> * DirectRunner doesn't specify a temp location directly, but test classes 
> will use a JUnit @Rule TemporaryDirectory. Individual tests set it as 
> necessary for tempLocation, and set a fake gs:// path for individual GCP IO 
> tests.
> * DataflowRunner tests pass an actual GCS path as tempRoot, and 
> TestDataflowRunner will initialize stagingLocation to this path.
> This setup makes it difficult to write RunnableOnService tests which pass for 
> both runners. We should separate temp location setup out of individual test 
> classes so that RunnableOnService tests "just work" on any runner.
> One solution would be to add logic inside TestPipeline#testingPipelineOptions:
> # If --tempRoot is specified, use it to set tempLocation and stagingLocation. 
> Otherwise, use a JUnit TemporaryDirectory to set it
> # If tempLocation is a GCS path, use it to set stagingLocation. Otherwise, 
> use a fake gcs path (i.e. gs://foo)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #753: [BEAM-436] Re-enable RunnableOnService tes...

[swegner]

Github user swegner closed the pull request at:

https://github.com/apache/incubator-beam/pull/753


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-630) DataflowRunnableOnService test suite is broken

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-630:


 Summary: DataflowRunnableOnService test suite is broken
 Key: BEAM-630
 URL: https://issues.apache.org/jira/browse/BEAM-630
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Eugene Kirpichov


Looks like it's related to problems with the new DoFnSignatures/Invoker: 
https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1134/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1322

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1322

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1140

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1321

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Port DataflowJavaSDK-351 to Beam

[lcwik]

Port DataflowJavaSDK-351 to Beam

This closes #952


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0800a8cb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0800a8cb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0800a8cb

Branch: refs/heads/master
Commit: 0800a8cb2b33856e5b0c93d669046648ebe7c01b
Parents: 84e8bfb 9febae7
Author: Luke Cwik 
Authored: Tue Sep 13 11:17:19 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 11:17:19 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  8 +--
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  7 +++
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 46 
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 42 ++-
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 56 
 5 files changed, 152 insertions(+), 7 deletions(-)
--

[1/2] incubator-beam git commit: Forward port DataflowJavaSDK-351 to Beam

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 84e8bfb13 -> 0800a8cb2


Forward port DataflowJavaSDK-351 to Beam


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9febae72
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9febae72
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9febae72

Branch: refs/heads/master
Commit: 9febae72666ba23cedbfbefdf2a19f6b9d6f14bf
Parents: 84e8bfb
Author: Pei He 
Authored: Mon Sep 12 21:02:35 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 11:16:56 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  8 +--
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  7 +++
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 46 
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 42 ++-
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 56 
 5 files changed, 152 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9febae72/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 1306e59..91f6073 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -390,10 +390,6 @@ public class BigQueryIO {
   "Validation of query \"%1$s\" failed. If the query depends on an 
earlier stage of the"
   + " pipeline, This validation can be disabled using 
#withoutValidation.";
 
-  // The maximum number of retries to poll a BigQuery job in the cleanup 
phase.
-  // We expect the jobs have already DONE, and don't need a high max 
retires.
-  private static final int CLEANUP_JOB_POLL_MAX_RETRIES = 10;
-
   private Bound() {
 this(
 null /* name */,
@@ -582,8 +578,8 @@ public class BigQueryIO {
 JobReference jobRef = new JobReference()
 .setProjectId(executingProject)
 .setJobId(getExtractJobId(jobIdToken));
-Job extractJob = bqServices.getJobService(bqOptions).pollJob(
-jobRef, CLEANUP_JOB_POLL_MAX_RETRIES);
+Job extractJob = bqServices.getJobService(bqOptions)
+.getJob(jobRef);
 
 Collection extractFiles = null;
 if (extractJob != null) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9febae72/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
index c0951fc..16b3a39 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
@@ -99,6 +99,13 @@ interface BigQueryServices extends Serializable {
  */
 JobStatistics dryRunQuery(String projectId, String query)
 throws InterruptedException, IOException;
+
+/**
+ * Gets the specified {@link Job} by the given {@link JobReference}.
+ *
+ * Returns null if the job is not found.
+ */
+Job getJob(JobReference jobRef) throws IOException, InterruptedException;
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9febae72/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
index 20dadff..7d98401 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
@@ -283,6 +283,52 @@ class BigQueryServicesImpl 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1139

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1138

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Port DataflowJavaSDK-337 to Beam

[lcwik]

Port DataflowJavaSDK-337 to Beam

This closes #950


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/84e8bfb1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/84e8bfb1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/84e8bfb1

Branch: refs/heads/master
Commit: 84e8bfb1344d6b6130241c03214bd0550b4fead5
Parents: b304d03 31bd5ba
Author: Luke Cwik 
Authored: Tue Sep 13 11:01:36 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 13 11:01:36 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 16 ++
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 57 ++--
 2 files changed, 58 insertions(+), 15 deletions(-)
--

[1/2] incubator-beam git commit: Forward port DataflowJavaSDK-337 to Beam

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b304d037f -> 84e8bfb13


Forward port DataflowJavaSDK-337 to Beam


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/31bd5ba4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/31bd5ba4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/31bd5ba4

Branch: refs/heads/master
Commit: 31bd5ba4bc44862c8042cdf6a79be8a6ebda49a4
Parents: b304d03
Author: Pei He 
Authored: Mon Sep 12 20:38:28 2016 -0700
Committer: Pei He 
Committed: Mon Sep 12 20:49:50 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 16 ++
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 57 ++--
 2 files changed, 58 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/31bd5ba4/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 6dde581..1306e59 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -481,6 +481,22 @@ public class BigQueryIO {
 // read is properly specified.
 BigQueryOptions bqOptions = 
input.getPipeline().getOptions().as(BigQueryOptions.class);
 
+String tempLocation = bqOptions.getTempLocation();
+checkArgument(
+!Strings.isNullOrEmpty(tempLocation),
+"BigQueryIO.Read needs a GCS temp location to store temp files.");
+if (bigQueryServices == null) {
+  try {
+GcsPath.fromUri(tempLocation);
+  } catch (IllegalArgumentException e) {
+throw new IllegalArgumentException(
+String.format(
+"BigQuery temp location expected a valid 'gs://' path, but 
was given '%s'",
+tempLocation),
+e);
+  }
+}
+
 TableReference table = getTableWithDefaultProject(bqOptions);
 if (table == null && query == null) {
   throw new IllegalStateException(

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/31bd5ba4/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
index 57eb4ff..19eeca5 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
@@ -360,8 +360,6 @@ public class BigQueryIOTest implements Serializable {
   @Mock private transient IOChannelFactory mockIOChannelFactory;
   @Mock(extraInterfaces = Serializable.class) private transient DatasetService 
mockDatasetService;
 
-  private transient BigQueryOptions bqOptions;
-
   private void checkReadTableObject(
   BigQueryIO.Read.Bound bound, String project, String dataset, String 
table) {
 checkReadTableObjectWithValidate(bound, project, dataset, table, true);
@@ -410,10 +408,6 @@ public class BigQueryIOTest implements Serializable {
 
   @Before
   public void setUp() throws IOException {
-bqOptions = 
TestPipeline.testingPipelineOptions().as(BigQueryOptions.class);
-bqOptions.setProject("defaultProject");
-
bqOptions.setTempLocation(testFolder.newFolder("BigQueryIOTest").getAbsolutePath());
-
 MockitoAnnotations.initMocks(this);
   }
 
@@ -468,8 +462,9 @@ public class BigQueryIOTest implements Serializable {
   public void testValidateReadSetsDefaultProject() throws Exception {
 String projectId = "someproject";
 String datasetId = "somedataset";
-BigQueryOptions options = 
TestPipeline.testingPipelineOptions().as(BigQueryOptions.class);
-options.setProject(projectId);
+BigQueryOptions bqOptions = 
PipelineOptionsFactory.as(BigQueryOptions.class);
+bqOptions.setProject(projectId);
+bqOptions.setTempLocation("gs://testbucket/testdir");
 
 FakeBigQueryServices fakeBqServices = new FakeBigQueryServices()
 .withJobService(mockJobService)
@@ -477,7 +472,7 @@ public class 

[jira] [Commented] (BEAM-625) Make Dataflow Python Materialized PCollection representation more efficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487766#comment-15487766
 ] 

ASF GitHub Bot commented on BEAM-625:
-

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/946


> Make Dataflow Python Materialized PCollection representation more efficient
> ---
>
> Key: BEAM-625
> URL: https://issues.apache.org/jira/browse/BEAM-625
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Konstantinos Katsiapis
>Assignee: Frances Perry
>
> This will be a several step process which will involve adding better support 
> for compression as well as Avro.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #946: [BEAM-625] Making Dataflow Python Material...

[katsiapis]

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/946


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #946

[dhalperi]

Closes #946


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4b584ca2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4b584ca2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4b584ca2

Branch: refs/heads/python-sdk
Commit: 4b584ca26b12f42a21b25b7d72c2e57b20a1e7ad
Parents: bc32bc8 f5557c0
Author: Dan Halperin 
Authored: Tue Sep 13 09:59:56 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 13 09:59:56 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py  | 252 +
 sdks/python/apache_beam/io/avroio_test.py |  88 +++--
 2 files changed, 202 insertions(+), 138 deletions(-)
--

[1/2] incubator-beam git commit: Refactoring code in avroio.py to allow for re-use.

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk bc32bc866 -> 4b584ca26


Refactoring code in avroio.py to allow for re-use.

* Making sure that _AvroUtils validates the sync_marker.
* This should detect corrupted or not-properly formatted AVRO files.
* Simplifying block reading.
* Running snappy tests only when snappy is installed in the system.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5557c00
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5557c00
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5557c00

Branch: refs/heads/python-sdk
Commit: f5557c00cd46c8560cad5027678603056a6235ff
Parents: bc32bc8
Author: Gus Katsiapis 
Authored: Mon Sep 12 10:11:44 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 13 09:59:17 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py  | 252 +
 sdks/python/apache_beam/io/avroio_test.py |  88 +++--
 2 files changed, 202 insertions(+), 138 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5557c00/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 7ad3842..196e760 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -14,7 +14,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-
 """Implements a source for reading Avro files."""
 
 import os
@@ -22,7 +21,7 @@ import StringIO
 import zlib
 
 from avro import datafile
-from avro import io as avro_io
+from avro import io as avroio
 from avro import schema
 
 from apache_beam.io import filebasedsource
@@ -75,159 +74,170 @@ class ReadFromAvro(PTransform):
 self._min_bundle_size = min_bundle_size
 
   def apply(self, pcoll):
-return pcoll.pipeline | Read(_AvroSource(
-file_pattern=self._file_pattern, 
min_bundle_size=self._min_bundle_size))
+return pcoll.pipeline | Read(
+_AvroSource(
+file_pattern=self._file_pattern,
+min_bundle_size=self._min_bundle_size))
 
 
-class _AvroSource(filebasedsource.FileBasedSource):
-  """A source for reading Avro files.
+class _AvroUtils(object):
 
-  ``_AvroSource`` is implemented using the file-based source framework 
available
-  in module 'filebasedsource'. Hence please refer to module 'filebasedsource'
-  to fully understand how this source implements operations common to all
-  file-based sources such as file-pattern expansion and splitting into bundles
-  for parallel processing.
-  """
+  @staticmethod
+  def read_meta_data_from_file(f):
+"""Reads metadata from a given Avro file.
 
-  def __init__(self, file_pattern, min_bundle_size=0):
-super(_AvroSource, self).__init__(file_pattern, min_bundle_size)
-self._avro_schema = None
-self._codec = None
-self._sync_marker = None
-
-  class AvroBlock(object):
-"""Represents a block of an Avro file."""
-
-def __init__(self, block_bytes, num_records, avro_schema, avro_codec,
- offset):
-  self._block_bytes = block_bytes
-  self._num_records = num_records
-  self._avro_schema = avro_schema
-  self._avro_codec = avro_codec
-  self._offset = offset
-
-def size(self):
-  return len(self._block_bytes)
-
-def _decompress_bytes(self, encoding, data):
-  if encoding == 'null':
-return data
-  elif encoding == 'deflate':
-# zlib.MAX_WBITS is the window size. '-' sign indicates that this is
-# raw data (without headers). See zlib and Avro documentations for more
-# details.
-return zlib.decompress(data, -zlib.MAX_WBITS)
-  else:
-raise ValueError('Unsupported compression type: %r', encoding)
+Args:
+  f: Avro file to read.
+Returns:
+  a tuple containing the codec, schema, and the sync marker of the Avro
+  file.
 
-def records(self):
-  decompressed_bytes = self._decompress_bytes(self._avro_codec,
-  self._block_bytes)
-  decoder = avro_io.BinaryDecoder(StringIO.StringIO(decompressed_bytes))
-  reader = avro_io.DatumReader(
-  writers_schema=schema.parse(self._avro_schema),
-  readers_schema=schema.parse(self._avro_schema))
+Raises:
+  ValueError: if the file does not start with the byte sequence defined in
+  the specification.
+"""
+if f.tell() > 0:
+  f.seek(0)
+decoder = avroio.BinaryDecoder(f)
+header = avroio.DatumReader().read_data(datafile.META_SCHEMA,
+ 

[jira] [Updated] (BEAM-629) Cleanup SDK regardign lint warnings (and lint warning suppression).

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-629:

Component/s: sdk-py

> Cleanup SDK regardign lint warnings (and lint warning suppression).
> ---
>
> Key: BEAM-629
> URL: https://issues.apache.org/jira/browse/BEAM-629
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Konstantinos Katsiapis
>Priority: Minor
>
> As discussed in https://github.com/apache/incubator-beam/pull/946



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-629) Cleanup SDK regardign lint warnings (and lint warning suppression).

[Konstantinos Katsiapis (JIRA)]

Konstantinos Katsiapis created BEAM-629:
---

 Summary: Cleanup SDK regardign lint warnings (and lint warning 
suppression).
 Key: BEAM-629
 URL: https://issues.apache.org/jira/browse/BEAM-629
 Project: Beam
  Issue Type: Improvement
Reporter: Konstantinos Katsiapis
Priority: Minor


As discussed in https://github.com/apache/incubator-beam/pull/946



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1136

[Apache Jenkins Server]

See