Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #507

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #438: Temp location

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/438

Temp location




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam temp-location

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/438.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #438


commit 34d2055910f561c8d37d3bd4c3b406e783232f3a
Author: Pei He 
Date:   2016-06-08T23:46:14Z

Remove defaulting tempLocation to stagingLocation

commit 14f2a6f0f863ade9fb5bad104e984355769a0f50
Author: Pei He 
Date:   2016-06-08T23:47:05Z

Add GcpTempLocation, and use it as the default stagingLocation

commit b51c56630a6c346650dbb61ad429f22e048c9108
Author: Pei He 
Date:   2016-06-08T23:48:14Z

Add BigQueryTempLocation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-317) Implement PAssert without using Side Inputs

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-317:


Assignee: Kenneth Knowles  (was: Thomas Groh)

> Implement PAssert without using Side Inputs
> ---
>
> Key: BEAM-317
> URL: https://issues.apache.org/jira/browse/BEAM-317
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Minor
>
> SideInputs are not supported equally by all runners. Implementing assertions 
> as a main input and a serialized assertion allows runner implementors to test 
> their implementations prior to constructing side inputs.
> satisfies is the same, but accumulates the entire input into a single pane 
> which it takes as a main input.
> containsInAnyOrder encodes the expected elements as they need not be 
> serializable, and then can be implemented in terms of satisfies.
> empty can be implemented by failing if provided with any elements, so long as 
> the success counter is not the means by which success is measured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-317) Implement PAssert without using Side Inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321656#comment-15321656
 ] 

ASF GitHub Bot commented on BEAM-317:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/437

[BEAM-317] Base PAssert on GBK instead of side inputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Previously `PAssert` - hence all `RunnableOnService`/`NeedsRunner` tests - 
required side input support. This created a very steep on ramp for new runners.

`GroupByKey` is a bit more fundamental and most backends will be able to 
group by key in the global window very quickly. So switching the primitive used 
to gather all the contents of a `PCollection` for assertions should make it a 
bit easier to get early feedback during runner development.

This exposed issues in a test, which is fixed in a separate commit, should 
you choose to review them separately.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam PAssert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit 239f9f93c17d87b45e1bd7750f483cd2966a9ae8
Author: Kenneth Knowles 
Date:   2016-06-08T22:07:52Z

Base PAssert on GBK instead of side inputs

Previously PAssert - hence all RunnableOnService/NeedsRunner
tests - required side input support. This created a very steep
on ramp for new runners.

GroupByKey is a bit more fundamental and most backends will be
able to group by key in the global window very quickly. So switching
the primitive used to gather all the contents of a PCollection for
assertions should make it a bit easier to get early feedback during
runner development.

commit c17cec636632cdeb5f208bbf2655456b54acdc0f
Author: Kenneth Knowles 
Date:   2016-06-08T23:22:34Z

Fix TriggerExampleTest




> Implement PAssert without using Side Inputs
> ---
>
> Key: BEAM-317
> URL: https://issues.apache.org/jira/browse/BEAM-317
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Minor
>
> SideInputs are not supported equally by all runners. Implementing assertions 
> as a main input and a serialized assertion allows runner implementors to test 
> their implementations prior to constructing side inputs.
> satisfies is the same, but accumulates the entire input into a single pane 
> which it takes as a main input.
> containsInAnyOrder encodes the expected elements as they need not be 
> serializable, and then can be implemented in terms of satisfies.
> empty can be implemented by failing if provided with any elements, so long as 
> the success counter is not the means by which success is measured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #506

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-159) Support fixed number of shards in custom sinks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321532#comment-15321532
 ] 

ASF GitHub Bot commented on BEAM-159:
-

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/436

Raise error rather than overwrite data when sharding is not respected

This is particularly relevant when withoutSharding() is used due to
[BEAM-159].

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam sharding

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #436


commit aa27f7fde7063a55b566b5a0a28c0b3bb667e9fe
Author: Robert Bradshaw 
Date:   2016-06-08T21:59:56Z

Raise error rather than overwrite data when sharding is not respected

This is particularly relevant when withoutSharding() is used due to
[BEAM-159].




> Support fixed number of shards in custom sinks
> --
>
> Key: BEAM-159
> URL: https://issues.apache.org/jira/browse/BEAM-159
> Project: Beam
>  Issue Type: Improvement
>Reporter: Eugene Kirpichov
>
> TextIO supports .withNumShards, however custom sinks, in particular 
> FileBasedSinks, provide no support for controlling sharding. Some users want 
> this, e.g. 
> http://stackoverflow.com/questions/36316304/set-num-of-output-shard-in-write-tosink-in-dataflow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #504

[Apache Jenkins Server]

See 

incubator-beam git commit: [maven-release-plugin] prepare release v0.1.0-incubating-RC3

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.1.0-incubating 8bf311932 -> 99a04ab8b


[maven-release-plugin] prepare release v0.1.0-incubating-RC3


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99a04ab8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99a04ab8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/99a04ab8

Branch: refs/heads/release-0.1.0-incubating
Commit: 99a04ab8b0107fbfb40d796d51df3d77aba914ba
Parents: 8bf3119
Author: Davor Bonaci 
Authored: Wed Jun 8 13:04:25 2016 -0700
Committer: Davor Bonaci 
Committed: Wed Jun 8 13:04:25 2016 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/examples/pom.xml  | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/flink/runner/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hdfs/pom.xml   | 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 26 files changed, 27 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 871abcd..d5fab50 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index ae3159e..9f90673 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 384090b..0a36801 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0a2cd55..4f21eb2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.1.0-incubating-SNAPSHOT
+  0.1.0-incubating
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-release-0.1.0-incubating
+v0.1.0-incubating-RC3
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index adb1168..a9ef3c9 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-runners-parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99a04ab8/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml

[jira] [Commented] (BEAM-240) Add display data link URLs for sources / sinks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321339#comment-15321339
 ] 

ASF GitHub Bot commented on BEAM-240:
-

Github user swegner closed the pull request at:

https://github.com/apache/incubator-beam/pull/300


> Add display data link URLs for sources / sinks
> --
>
> Key: BEAM-240
> URL: https://issues.apache.org/jira/browse/BEAM-240
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #389: Test case for package-private PipelineOpti...

[swegner]

Github user swegner closed the pull request at:

https://github.com/apache/incubator-beam/pull/389


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321326#comment-15321326
 ] 

ASF GitHub Bot commented on BEAM-22:


GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/435

[BEAM-22] Shutdown the InProcessPipelineRunner after Terminating Abnormally

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Ensure that the executor service is shutdown, and the monitor is not
rescheduled, after an exception is thrown.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
shutdown_executor_after_exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #435


commit 591134749f2d24a7d6550e0bb00845a36cdb1616
Author: Thomas Groh 
Date:   2016-06-08T19:00:05Z

Shutdown the InProcessPipelineRunner after Terminating Abnormally

Ensure that the executor service is shutdown, and the monitor is not
rescheduled, after an exception is thrown.




> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #435: [BEAM-22] Shutdown the InProcessPipelineRu...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/435

[BEAM-22] Shutdown the InProcessPipelineRunner after Terminating Abnormally

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Ensure that the executor service is shutdown, and the monitor is not
rescheduled, after an exception is thrown.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
shutdown_executor_after_exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #435


commit 591134749f2d24a7d6550e0bb00845a36cdb1616
Author: Thomas Groh 
Date:   2016-06-08T19:00:05Z

Shutdown the InProcessPipelineRunner after Terminating Abnormally

Ensure that the executor service is shutdown, and the monitor is not
rescheduled, after an exception is thrown.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-288) Improve javadoc for UnboundedSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321321#comment-15321321
 ] 

ASF GitHub Bot commented on BEAM-288:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/340


> Improve javadoc for UnboundedSource
> ---
>
> Key: BEAM-288
> URL: https://issues.apache.org/jira/browse/BEAM-288
> Project: Beam
>  Issue Type: Bug
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> While working on the pubsub source noticed the UnboundedSource and associated 
> Reader / Checkpoint API needed some important clarrification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Merge branch 'master' into release-0.1.0-incubating

[dhalperi]

Merge branch 'master' into release-0.1.0-incubating


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8bf31193
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8bf31193
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8bf31193

Branch: refs/heads/release-0.1.0-incubating
Commit: 8bf311932996d046b678daf7e0ade2967a6a3b8f
Parents: 7e5d2f9 ccc1e81
Author: Dan Halperin 
Authored: Wed Jun 8 12:34:50 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 12:34:50 2016 -0700

--
 distributions/pom.xml   | 62 
 distributions/src/main/assembly/src.xml | 55 
 pom.xml |  7 ++--
 3 files changed, 3 insertions(+), 121 deletions(-)
--

[1/2] incubator-beam git commit: Use Apache's release to produce the source release

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3f3dffcb6 -> 7fb21f223


Use Apache's release to produce the source release

And use final-name to override the artifact and directory names


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70e275b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70e275b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70e275b7

Branch: refs/heads/master
Commit: 70e275b7683331005a2c398a606a2785ed86aa24
Parents: 3f3dffc
Author: Dan Halperin 
Authored: Wed Jun 8 11:35:19 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 11:59:14 2016 -0700

--
 distributions/pom.xml   | 62 
 distributions/src/main/assembly/src.xml | 55 
 pom.xml |  7 ++--
 3 files changed, 3 insertions(+), 121 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/70e275b7/distributions/pom.xml
--
diff --git a/distributions/pom.xml b/distributions/pom.xml
deleted file mode 100644
index cb3bd66..000
--- a/distributions/pom.xml
+++ /dev/null
@@ -1,62 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-
-  4.0.0
-
-  
-org.apache.beam
-beam-parent
-0.2.0-incubating-SNAPSHOT
-../pom.xml
-  
-
-  apache-beam
-  pom
-  Apache Beam :: Distributions
-
-  
-
-  release
-  
-
-  
-org.apache.maven.plugins
-maven-assembly-plugin
-
-  
-src
-package
-
-  single
-
-
-  apache-beam-${project.version}
-  
-src/main/assembly/src.xml
-  
-
-  
-
-  
-
-  
-
-  
-
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/70e275b7/distributions/src/main/assembly/src.xml
--
diff --git a/distributions/src/main/assembly/src.xml 
b/distributions/src/main/assembly/src.xml
deleted file mode 100644
index a0d7a87..000
--- a/distributions/src/main/assembly/src.xml
+++ /dev/null
@@ -1,55 +0,0 @@
-
-
-http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
-  
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3
 http://maven.apache.org/xsd/assembly-1.1.3.xsd;>
-
-  src
-
-  
-zip
-  
-
-  
-
-  ${basedir}/..
-  /
-  
-
-
-**/target/**
-**/bin/**
-
-**/.idea/**
-**/*.iml
-**/*.ipr
-**/*.iws
-
-**/.classpath
-**/.project
-**/.factorypath
-**/.checkstyle
-**/.settings/**
-
-**/dependency-reduced-pom.xml
-
-**/*~
-  
-
-  
-
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/70e275b7/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 77a0d1a..e7cc5f7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -140,7 +140,6 @@
   build order. -->
 sdks/java/maven-archetypes
 examples
-distributions
   
 
   
@@ -922,8 +921,8 @@
   
 
 
-
+
 
   org.apache.maven.plugins
   maven-assembly-plugin
@@ -931,7 +930,7 @@
 
   source-release-assembly
   
-true
+apache-beam-${project.version}
   
 
   

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #503

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #434: Use Apache's release to produce the source...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/434

Use Apache's release to produce the source release

And use final-name to override the artifact and directory names

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam mymaster

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #434


commit 70e275b7683331005a2c398a606a2785ed86aa24
Author: Dan Halperin 
Date:   2016-06-08T18:35:19Z

Use Apache's release to produce the source release

And use final-name to override the artifact and directory names




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] incubator-beam git commit: Merge branch 'master' into release-0.1.0-incubating

[dhalperi]

Merge branch 'master' into release-0.1.0-incubating


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/492d919f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/492d919f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/492d919f

Branch: refs/heads/release-0.1.0-incubating
Commit: 492d919f38d2211f78e808b8a476138e691fcf96
Parents: 347d159 35098dc
Author: Dan Halperin 
Authored: Wed Jun 8 10:38:32 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 10:38:32 2016 -0700

--
 distributions/src/main/assembly/src.xml |  2 +-
 runners/direct-java/pom.xml |  4 
 runners/flink/examples/pom.xml  | 13 +
 runners/flink/pom.xml   | 15 ---
 runners/flink/runner/pom.xml| 12 
 runners/spark/pom.xml   |  4 
 6 files changed, 34 insertions(+), 16 deletions(-)
--

[3/4] incubator-beam git commit: Fix pom shading of Guava

[dhalperi]

Fix pom shading of Guava

There are two packages: com.google.common and com.google.thirdparty.
Both need to be repackaged/relocated, but we only had com.google.common
in a few places.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/35098dc6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/35098dc6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/35098dc6

Branch: refs/heads/release-0.1.0-incubating
Commit: 35098dc6971ad0a195dcfbd23979dc82d482b2c9
Parents: 340ad4a
Author: Dan Halperin 
Authored: Tue Jun 7 23:26:46 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 10:35:20 2016 -0700

--
 runners/direct-java/pom.xml | 4 
 runners/spark/pom.xml   | 4 
 2 files changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/35098dc6/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 98b0e20..0a248d3 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -201,6 +201,10 @@
   com.google.common
   
org.apache.beam.runners.direct.repackaged.com.google.common
 
+
+  com.google.thirdparty
+  
org.apache.beam.runners.direct.repackaged.com.google.thirdparty
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/35098dc6/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 1989d5f..b626b9f 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -187,6 +187,10 @@
 com.google.common
 
org.apache.beam.spark.relocated.com.google.common
   
+  
+com.google.thirdparty
+
org.apache.beam.spark.relocated.com.google.thirdparty
+  
 
 true
 spark-app

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #502

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-132) Release process for Beam artifacts to Maven

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321007#comment-15321007
 ] 

ASF GitHub Bot commented on BEAM-132:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/433


> Release process for Beam artifacts to Maven
> ---
>
> Key: BEAM-132
> URL: https://issues.apache.org/jira/browse/BEAM-132
> Project: Beam
>  Issue Type: New Feature
>  Components: project-management
>Reporter: Kostas Kloudas
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #433: [BEAM-132] pom.xml: skip source-release as...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/433


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-132] pom.xml: skip source-release assembly

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ce87f377b -> 3f3dffcb6


[BEAM-132] pom.xml: skip source-release assembly

We do this separately via our own assembly.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2ab4ebf9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2ab4ebf9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2ab4ebf9

Branch: refs/heads/master
Commit: 2ab4ebf9632cf82365329a198f52b31fb99a47eb
Parents: ce87f37
Author: Dan Halperin 
Authored: Wed Jun 8 10:07:35 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 10:07:35 2016 -0700

--
 pom.xml | 15 +++
 1 file changed, 15 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2ab4ebf9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 07977cf..77a0d1a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -921,6 +921,21 @@
 true
   
 
+
+
+
+  org.apache.maven.plugins
+  maven-assembly-plugin
+  
+
+  source-release-assembly
+  
+true
+  
+
+  
+
   
 
 

[2/2] incubator-beam git commit: Closes #433

[dhalperi]

Closes #433


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3f3dffcb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3f3dffcb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3f3dffcb

Branch: refs/heads/master
Commit: 3f3dffcb6bad737ed487712c4db342e5d0be08e6
Parents: ce87f37 2ab4ebf
Author: Dan Halperin 
Authored: Wed Jun 8 10:26:04 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jun 8 10:26:04 2016 -0700

--
 pom.xml | 15 +++
 1 file changed, 15 insertions(+)
--

[GitHub] incubator-beam pull request #433: [BEAM-132] pom.xml: skip source-release as...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/433

[BEAM-132] pom.xml: skip source-release assembly

We do this separately via our own assembly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam disable-assembly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #433


commit 2ab4ebf9632cf82365329a198f52b31fb99a47eb
Author: Dan Halperin 
Date:   2016-06-08T17:07:35Z

[BEAM-132] pom.xml: skip source-release assembly

We do this separately via our own assembly.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix pom shading of Guava

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9a3896d69 -> ce87f377b


Fix pom shading of Guava

There are two packages: com.google.common and com.google.thirdparty.
Both need to be repackaged/relocated, but we only had com.google.common
in a few places.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e0035936
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e0035936
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e0035936

Branch: refs/heads/master
Commit: e003593612e28465f515e8399b4c4827d50fe787
Parents: 9a3896d
Author: Dan Halperin 
Authored: Tue Jun 7 23:26:46 2016 -0700
Committer: Davor Bonaci 
Committed: Wed Jun 8 09:16:00 2016 -0700

--
 runners/direct-java/pom.xml | 4 
 runners/spark/pom.xml   | 4 
 2 files changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e0035936/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index f59fcb9..fb2d2e1 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -201,6 +201,10 @@
   com.google.common
   
org.apache.beam.runners.direct.repackaged.com.google.common
 
+
+  com.google.thirdparty
+  
org.apache.beam.runners.direct.repackaged.com.google.thirdparty
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e0035936/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index a3aadfb..4110689 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -187,6 +187,10 @@
 com.google.common
 
org.apache.beam.spark.relocated.com.google.common
   
+  
+com.google.thirdparty
+
org.apache.beam.spark.relocated.com.google.thirdparty
+  
 
 true
 spark-app

[2/2] incubator-beam git commit: This closes #431

[davor]

This closes #431


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ce87f377
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ce87f377
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ce87f377

Branch: refs/heads/master
Commit: ce87f377b4b643444087a25f05d730649e460d2d
Parents: 9a3896d e003593
Author: Davor Bonaci 
Authored: Wed Jun 8 09:16:07 2016 -0700
Committer: Davor Bonaci 
Committed: Wed Jun 8 09:16:07 2016 -0700

--
 runners/direct-java/pom.xml | 4 
 runners/spark/pom.xml   | 4 
 2 files changed, 8 insertions(+)
--

[1/2] incubator-beam git commit: Remove Unnecessary Fn cloning

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cc448e976 -> 9a3896d69


Remove Unnecessary Fn cloning

The ParDoInProcessEvaluator is provided clones of a DoFn when
appropriate, and should not serialize them.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0cf17302
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0cf17302
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0cf17302

Branch: refs/heads/master
Commit: 0cf17302f605aefde22c48e65e17da6e3cfc6238
Parents: 2173000
Author: Thomas Groh 
Authored: Fri Jun 3 17:57:24 2016 -0700
Committer: Thomas Groh 
Committed: Tue Jun 7 14:20:42 2016 -0700

--
 .../org/apache/beam/runners/direct/ParDoInProcessEvaluator.java  | 3 +--
 .../apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java  | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0cf17302/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoInProcessEvaluator.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoInProcessEvaluator.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoInProcessEvaluator.java
index 2cdf6cb..a3e2f18 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoInProcessEvaluator.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoInProcessEvaluator.java
@@ -27,7 +27,6 @@ import org.apache.beam.sdk.util.DoFnRunners;
 import org.apache.beam.sdk.util.DoFnRunners.OutputManager;
 import org.apache.beam.sdk.util.PushbackSideInputDoFnRunner;
 import org.apache.beam.sdk.util.ReadyCheckingSideInputReader;
-import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.common.CounterSet;
@@ -74,7 +73,7 @@ class ParDoInProcessEvaluator implements 
TransformEvaluator {
 DoFnRunner underlying =
 DoFnRunners.createDefault(
 evaluationContext.getPipelineOptions(),
-SerializableUtils.clone(fn),
+fn,
 sideInputReader,
 BundleOutputManager.create(outputBundles),
 mainOutputTag,

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0cf17302/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java
index 1127ed2..0f7afa1 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java
@@ -128,7 +128,7 @@ public class ParDoInProcessEvaluatorTest {
 Matchers.containsInAnyOrder(
 second, WindowedValue.of(1, new Instant(2468L), nonGlobalWindow, 
PaneInfo.NO_FIRING)));
 assertThat(result.getOutputBundles(), 
Matchers.contains(outputBundle));
-assertThat(RecorderFn.processed, containsInAnyOrder(1, 3));
+assertThat(fn.processed, containsInAnyOrder(1, 3));
 assertThat(
 
Iterables.getOnlyElement(result.getOutputBundles()).commit(Instant.now()).getElements(),
 Matchers.containsInAnyOrder(
@@ -170,7 +170,7 @@ public class ParDoInProcessEvaluatorTest {
   }
 
   private static class RecorderFn extends DoFn {
-private static Collection processed;
+private Collection processed;
 private final PCollectionView view;
 
 public RecorderFn(PCollectionView view) {

[GitHub] incubator-beam pull request #418: [BEAM-22] Remove unneccesary Fn cloning

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/418


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #418

[kenn]

This closes #418


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9a3896d6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9a3896d6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9a3896d6

Branch: refs/heads/master
Commit: 9a3896d697df7862641117a56001be5a0b44771d
Parents: cc448e9 0cf1730
Author: Kenneth Knowles 
Authored: Wed Jun 8 09:15:00 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Jun 8 09:15:00 2016 -0700

--
 .../org/apache/beam/runners/direct/ParDoInProcessEvaluator.java  | 3 +--
 .../apache/beam/runners/direct/ParDoInProcessEvaluatorTest.java  | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)
--

[jira] [Commented] (BEAM-196) Pipeline options must be available Context in DoFn.startBundle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320833#comment-15320833
 ] 

ASF GitHub Bot commented on BEAM-196:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/432


> Pipeline options must be available Context in DoFn.startBundle
> --
>
> Key: BEAM-196
> URL: https://issues.apache.org/jira/browse/BEAM-196
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Mark Shields
>Assignee: Maximilian Michels
> Fix For: 0.1.0-incubating
>
>
> Our (not yet merged) Java Pubsub implementation has code like this in a DoFn:
> @Override
> public void startBundle(Context c) throws Exception {
>   Preconditions.checkState(pubsubClient == null);
>   pubsubClient = PubsubClient.newClient(transportType,
>   timestampLabel, idLabel, 
> c.getPipelineOptions().as(PubsubOptions.class));
>   super.startBundle(c);
> }
> This fails with NPE since the pipeline options are not conveyed via the 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-287) Add new beam groupIds

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320834#comment-15320834
 ] 

ASF GitHub Bot commented on BEAM-287:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/415


> Add new beam groupIds
> -
>
> Key: BEAM-287
> URL: https://issues.apache.org/jira/browse/BEAM-287
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.1.0-incubating
>
>
> Now, all beam modules use the same groupId: {{org.apache.beam}}.
> If it's convenient when using {{search.maven.org}}, browsing the repository 
> is not easy as all artifacts are in the same location.
> As proposed initially, I will introduce new groupIds (like 
> {{org.apache.beam.io}}, {{org.apache.beam.runner}}, etc). We discussed about 
> that with [~davor] last week.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/5] incubator-beam git commit: [BEAM-196] provide PipelineOptions in DoFn

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ffbfc66e1 -> cc448e976


[BEAM-196] provide PipelineOptions in DoFn

- fixes NPE when accessing the PipelineOptions
- adds a test to verify that the PipelineOptions are available


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eced106e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eced106e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/eced106e

Branch: refs/heads/master
Commit: eced106e50ddb257524a7826ab7d27254be89da8
Parents: d10ae23
Author: Maximilian Michels 
Authored: Tue Jun 7 13:57:33 2016 +0200
Committer: Maximilian Michels 
Committed: Wed Jun 8 15:19:50 2016 +0200

--
 .../streaming/FlinkAbstractParDoWrapper.java| 11 ++-
 .../beam/runners/flink/PipelineOptionsTest.java | 97 +++-
 2 files changed, 100 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eced106e/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
index 117303c..a935011 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.flink.translation.wrappers.streaming;
 
+import 
org.apache.beam.runners.flink.translation.utils.SerializedPipelineOptions;
 import 
org.apache.beam.runners.flink.translation.wrappers.SerializableFnAggregatorWrapper;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
@@ -37,6 +38,7 @@ import com.google.common.base.Preconditions;
 import org.apache.flink.api.common.accumulators.Accumulator;
 import org.apache.flink.api.common.accumulators.AccumulatorHelper;
 import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.configuration.Configuration;
 import org.apache.flink.util.Collector;
 import org.joda.time.Instant;
 import org.joda.time.format.PeriodFormat;
@@ -52,7 +54,7 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 
   private final DoFn doFn;
   private final WindowingStrategy windowingStrategy;
-  private transient PipelineOptions options;
+  private final SerializedPipelineOptions serializedPipelineOptions;
 
   private DoFnProcessContext context;
 
@@ -62,7 +64,7 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 Preconditions.checkNotNull(doFn);
 
 this.doFn = doFn;
-this.options = options;
+this.serializedPipelineOptions = new SerializedPipelineOptions(options);
 this.windowingStrategy = windowingStrategy;
   }
 
@@ -107,7 +109,8 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 
 private WindowedValue element;
 
-private DoFnProcessContext(DoFn function, 
Collector outCollector) {
+private DoFnProcessContext(DoFn function,
+  Collector outCollector) {
   function.super();
   super.setupDelegateAggregators();
 
@@ -156,7 +159,7 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 
 @Override
 public PipelineOptions getPipelineOptions() {
-  return options;
+  return serializedPipelineOptions.getPipelineOptions();
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eced106e/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
--
diff --git 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
index 464c6df..d571f31 100644
--- 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
+++ 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
@@ -18,14 +18,29 @@
 package org.apache.beam.runners.flink;
 
 import 
org.apache.beam.runners.flink.translation.utils.SerializedPipelineOptions;
+import 

[GitHub] incubator-beam pull request #415: [BEAM-287] adjust README to changed Maven ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/415


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/5] incubator-beam git commit: [BEAM-287] adjust README to changed Maven layout

[mxm]

[BEAM-287] adjust README to changed Maven layout


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1fd0dfc7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1fd0dfc7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1fd0dfc7

Branch: refs/heads/master
Commit: 1fd0dfc7ad61078259a5ad43b37d42b873d40090
Parents: ffbfc66
Author: Maximilian Michels 
Authored: Fri Jun 3 11:13:45 2016 +0200
Committer: Maximilian Michels 
Committed: Wed Jun 8 18:08:34 2016 +0200

--
 runners/flink/README.md | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1fd0dfc7/runners/flink/README.md
--
diff --git a/runners/flink/README.md b/runners/flink/README.md
index cce17a2..69e2abb 100644
--- a/runners/flink/README.md
+++ b/runners/flink/README.md
@@ -163,7 +163,7 @@ The contents of the root `pom.xml` should be slightly 
changed aftewards (explana
   
 
   org.apache.beam
-  flink-runner_2.10
+  beam-runners-flink_2.10
   0.2.0-incubating-SNAPSHOT
 
   
@@ -186,11 +186,6 @@ The contents of the root `pom.xml` should be slightly 
changed aftewards (explana
   
org.apache.beam.runners.flink.examples.WordCount
 
   
-  
-
-  org.apache.flink:*
-
-  
 
   
 

[2/5] incubator-beam git commit: [flink] improve lifecycle of ParDoBoundWrapper

[mxm]

[flink] improve lifecycle of ParDoBoundWrapper


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d10ae23c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d10ae23c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d10ae23c

Branch: refs/heads/master
Commit: d10ae23c9bc9529d04d02951bfed01bbf2957773
Parents: ffbfc66
Author: Maximilian Michels 
Authored: Mon Jun 6 12:40:50 2016 +0200
Committer: Maximilian Michels 
Committed: Wed Jun 8 15:19:50 2016 +0200

--
 .../streaming/FlinkAbstractParDoWrapper.java  | 18 +++---
 1 file changed, 11 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d10ae23c/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
index bb6ed67..117303c 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkAbstractParDoWrapper.java
@@ -66,15 +66,21 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 this.windowingStrategy = windowingStrategy;
   }
 
-  private void initContext(DoFn function, 
Collector outCollector) {
-if (this.context == null) {
-  this.context = new DoFnProcessContext(function, outCollector);
-}
+  @Override
+  public void open(Configuration parameters) throws Exception {
+this.doFn.startBundle(context);
+  }
+
+  @Override
+  public void close() throws Exception {
+this.doFn.finishBundle(context);
   }
 
   @Override
   public void flatMap(WindowedValue value, Collector 
out) throws Exception {
-this.initContext(doFn, out);
+if (this.context == null) {
+  this.context = new DoFnProcessContext(doFn, out);
+}
 
 // for each window the element belongs to, create a new copy here.
 Collection windows = value.getWindows();
@@ -90,9 +96,7 @@ public abstract class FlinkAbstractParDoWrapper extends RichFl
 
   private void processElement(WindowedValue value) throws Exception {
 this.context.setElement(value);
-this.doFn.startBundle(context);
 doFn.processElement(context);
-this.doFn.finishBundle(context);
   }
 
   private class DoFnProcessContext extends DoFn.ProcessContext {

[5/5] incubator-beam git commit: This closes #415

[mxm]

This closes #415


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cc448e97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cc448e97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cc448e97

Branch: refs/heads/master
Commit: cc448e976d46a8a9341445e152b3b31ea8968a56
Parents: f5583cf 1fd0dfc
Author: Maximilian Michels 
Authored: Wed Jun 8 18:09:03 2016 +0200
Committer: Maximilian Michels 
Committed: Wed Jun 8 18:09:03 2016 +0200

--
 runners/flink/README.md | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)
--

[3/5] incubator-beam git commit: This closes #432

[mxm]

This closes #432


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5583cfa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5583cfa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5583cfa

Branch: refs/heads/master
Commit: f5583cfad6c857d919995edd385adaf0f41fd676
Parents: ffbfc66 eced106
Author: Maximilian Michels 
Authored: Wed Jun 8 18:04:50 2016 +0200
Committer: Maximilian Michels 
Committed: Wed Jun 8 18:04:50 2016 +0200

--
 .../streaming/FlinkAbstractParDoWrapper.java| 29 +++---
 .../beam/runners/flink/PipelineOptionsTest.java | 97 +++-
 2 files changed, 111 insertions(+), 15 deletions(-)
--

[GitHub] incubator-beam pull request #432: [BEAM-196] Additional fix to ensure the Pi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/432


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #501

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #432: [BEAM-196] Additional fix to ensure the Pi...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/432

[BEAM-196] Additional fix to ensure the PipelineOptions are available in 
DoFns.

Fixes a NullPointException if `PipelineOptions` are acceessed inside a DoFn 
and backs it up with a test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-196

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #432


commit 05287c4ddcc38378e4e8cd921b6f6394e43eed75
Author: Maximilian Michels 
Date:   2016-06-07T11:57:33Z

[BEAM-196] fix NPE in ParDoWrapper

commit 595146d5759a6f631f057050cdbe71f9849035f9
Author: Maximilian Michels 
Date:   2016-06-08T10:40:50Z

[flink] improve lifecycle of ParDoBoundWrapper




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-196) Pipeline options must be available Context in DoFn.startBundle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320368#comment-15320368
 ] 

ASF GitHub Bot commented on BEAM-196:
-

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/432

[BEAM-196] Additional fix to ensure the PipelineOptions are available in 
DoFns.

Fixes a NullPointException if `PipelineOptions` are acceessed inside a DoFn 
and backs it up with a test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-196

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #432


commit 05287c4ddcc38378e4e8cd921b6f6394e43eed75
Author: Maximilian Michels 
Date:   2016-06-07T11:57:33Z

[BEAM-196] fix NPE in ParDoWrapper

commit 595146d5759a6f631f057050cdbe71f9849035f9
Author: Maximilian Michels 
Date:   2016-06-08T10:40:50Z

[flink] improve lifecycle of ParDoBoundWrapper




> Pipeline options must be available Context in DoFn.startBundle
> --
>
> Key: BEAM-196
> URL: https://issues.apache.org/jira/browse/BEAM-196
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Mark Shields
>Assignee: Maximilian Michels
> Fix For: 0.1.0-incubating
>
>
> Our (not yet merged) Java Pubsub implementation has code like this in a DoFn:
> @Override
> public void startBundle(Context c) throws Exception {
>   Preconditions.checkState(pubsubClient == null);
>   pubsubClient = PubsubClient.newClient(transportType,
>   timestampLabel, idLabel, 
> c.getPipelineOptions().as(PubsubOptions.class));
>   super.startBundle(c);
> }
> This fails with NPE since the pipeline options are not conveyed via the 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-196) Pipeline options must be available Context in DoFn.startBundle

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reopened BEAM-196:
-

Still an issue. Pull request coming up.

> Pipeline options must be available Context in DoFn.startBundle
> --
>
> Key: BEAM-196
> URL: https://issues.apache.org/jira/browse/BEAM-196
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Mark Shields
>Assignee: Maximilian Michels
> Fix For: 0.1.0-incubating
>
>
> Our (not yet merged) Java Pubsub implementation has code like this in a DoFn:
> @Override
> public void startBundle(Context c) throws Exception {
>   Preconditions.checkState(pubsubClient == null);
>   pubsubClient = PubsubClient.newClient(transportType,
>   timestampLabel, idLabel, 
> c.getPipelineOptions().as(PubsubOptions.class));
>   super.startBundle(c);
> }
> This fails with NPE since the pipeline options are not conveyed via the 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #500

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #431: Fix pom shading of Guava

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/431

Fix pom shading of Guava

There are two packages: com.google.common and com.google.thirdparty.
Both need to be repackaged/relocated, but we only had com.google.common
in a few places.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam shade-direct-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/431.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #431


commit aae39c3c980885576ad3b24d30e1705575619714
Author: Dan Halperin 
Date:   2016-06-08T06:26:46Z

Fix pom shading of Guava

There are two packages: com.google.common and com.google.thirdparty.
Both need to be repackaged/relocated, but we only had com.google.common
in a few places.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---