incubator-beam git commit: SparkRunner calls pipeline.run

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c314e670e -> ac0875de8


SparkRunner calls pipeline.run

* Remove SparkStreamingPipelineOptions.
* Run pipeline with Pipeline.run().
* Better EmbeddedKafka.
* Avoid NPE if factory wasn't created.
* Let EmbeddedKafka/Zookeeper discover ports on their own.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ac0875de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ac0875de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ac0875de

Branch: refs/heads/master
Commit: ac0875de84085e1298575d0887e83e5deee5f418
Parents: c314e67
Author: Sela 
Authored: Wed Jul 27 23:11:37 2016 +0300
Committer: Dan Halperin 
Committed: Wed Aug 3 22:31:48 2016 -0700

--
 .../runners/spark/SparkPipelineOptions.java |  6 
 .../apache/beam/runners/spark/SparkRunner.java  | 14 +++--
 .../runners/spark/SparkRunnerRegistrar.java |  6 ++--
 .../spark/SparkStreamingPipelineOptions.java| 32 
 .../beam/runners/spark/TestSparkRunner.java |  2 --
 .../apache/beam/runners/spark/DeDupTest.java|  2 +-
 .../beam/runners/spark/EmptyInputTest.java  |  2 +-
 .../beam/runners/spark/SimpleWordCountTest.java |  4 +--
 .../runners/spark/SparkRunnerRegistrarTest.java |  2 +-
 .../apache/beam/runners/spark/TfIdfTest.java|  2 +-
 .../beam/runners/spark/io/AvroPipelineTest.java |  2 +-
 .../beam/runners/spark/io/NumShardsTest.java|  2 +-
 .../io/hadoop/HadoopFileFormatPipelineTest.java |  2 +-
 .../spark/translation/CombineGloballyTest.java  |  2 +-
 .../spark/translation/CombinePerKeyTest.java|  2 +-
 .../spark/translation/DoFnOutputTest.java   |  6 ++--
 .../translation/MultiOutputWordCountTest.java   |  2 +-
 .../spark/translation/SerializationTest.java|  2 +-
 .../spark/translation/SideEffectsTest.java  |  8 ++---
 .../streaming/FlattenStreamingTest.java |  8 ++---
 .../streaming/KafkaStreamingTest.java   | 13 
 .../streaming/SimpleStreamingWordCountTest.java |  8 ++---
 .../streaming/utils/EmbeddedKafkaCluster.java   |  4 ++-
 23 files changed, 49 insertions(+), 84 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ac0875de/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 4bb2a57..6ef3741 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -33,4 +33,10 @@ public interface SparkPipelineOptions extends 
PipelineOptions, StreamingOptions,
   @Default.String("local[1]")
   String getSparkMaster();
   void setSparkMaster(String master);
+
+  @Description("Timeout to wait (in msec) for a streaming execution to stop, 
-1 runs until "
+  + "execution is stopped")
+  @Default.Long(-1)
+  Long getTimeout();
+  void setTimeout(Long batchInterval);
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ac0875de/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index dfda987..d994ec4 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -69,8 +69,6 @@ import org.slf4j.LoggerFactory;
  * options.setSparkMaster("spark://host:port");
  * EvaluationResult result = SparkRunner.create(options).run(p);
  * }
- *
- * To create a Spark streaming pipeline runner use {@link 
SparkStreamingPipelineOptions}
  */
 public final class SparkRunner extends PipelineRunner {
 
@@ -146,12 +144,6 @@ public final class SparkRunner extends 
PipelineRunner {
   @Override
   public EvaluationResult run(Pipeline pipeline) {
 try {
-  // validate streaming configuration
-  if (mOptions.isStreaming() && !(mOptions instanceof 
SparkStreamingPipelineOptions)) {
-throw new RuntimeException("A streaming job must be configured with "
-+ SparkStreamingPipelineOptions.class.getSimpleName() + ", found "
-+ mOptions.getClass().getSimpleName());
-  }
   LOG.info("Executing pipeline using the SparkRunner.");
   JavaSparkContext jsc = 

[GitHub] incubator-beam pull request #783: Run findbugs in the test-compile phase

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/783


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #783

[dhalperi]

Closes #783


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c314e670
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c314e670
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c314e670

Branch: refs/heads/master
Commit: c314e670e0113cddc40af680f5ce8a5134d61e9a
Parents: 9a329aa 14c6d99
Author: Dan Halperin 
Authored: Wed Aug 3 22:28:17 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 22:28:17 2016 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Run findbugs in the test-compile phase

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9a329aada -> c314e670e


Run findbugs in the test-compile phase


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/14c6d99e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/14c6d99e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/14c6d99e

Branch: refs/heads/master
Commit: 14c6d99e087b2e1606422821341136a5d5e8ec23
Parents: 9a329aa
Author: Kenneth Knowles 
Authored: Wed Aug 3 21:31:17 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 21:31:17 2016 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14c6d99e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 23b7e4d..9e58ffe 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1029,7 +1029,7 @@
 
   
 
-  test
+  test-compile
   
 check
   

[GitHub] incubator-beam pull request #783: Run findbugs in the test-compile phase

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/783

Run findbugs in the test-compile phase

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This should help some problems fail faster.

R: @dhalperi 
CC: @swegner @jasonkuster


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam findbugs-earlier

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/783.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #783


commit 14c6d99e087b2e1606422821341136a5d5e8ec23
Author: Kenneth Knowles 
Date:   2016-08-04T04:31:17Z

Run findbugs in the test-compile phase




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407076#comment-15407076
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/782

[BEAM-498] Port easy bits of the SDK to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam transforms-new-DoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/782.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #782


commit 3fd414f9dd2b2aeb41ab7a24b635adb234ec5056
Author: Kenneth Knowles 
Date:   2016-08-04T02:55:21Z

Port join library to new DoFn

commit ab22cd9933bd13571c03e7dd66e4d01a6e38d7f3
Author: Kenneth Knowles 
Date:   2016-08-04T02:56:33Z

Port mentions of OldDoFn in PipelineOptions

commit 866d2c7dda01e04ff040b2ed655e6390c6b56ef4
Author: Kenneth Knowles 
Date:   2016-08-04T03:15:12Z

Port easy Java SDK tests to new DoFn

commit 2504240de6115139addf051c354fae4b3c49b67c
Author: Kenneth Knowles 
Date:   2016-08-04T03:15:58Z

Port PAssert to new DoFn

commit d16cc7f7dc14eaab1564b42177980ada149f7f99
Author: Kenneth Knowles 
Date:   2016-08-04T03:22:26Z

Port easy I/O transforms to new DoFn

commit 3f949838df812012175a086c287e99c25bca894e
Author: Kenneth Knowles 
Date:   2016-08-04T03:27:28Z

Port easy transforms to new DoFn




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #782: [BEAM-498] Port easy bits of the SDK to ne...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/782

[BEAM-498] Port easy bits of the SDK to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam transforms-new-DoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/782.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #782


commit 3fd414f9dd2b2aeb41ab7a24b635adb234ec5056
Author: Kenneth Knowles 
Date:   2016-08-04T02:55:21Z

Port join library to new DoFn

commit ab22cd9933bd13571c03e7dd66e4d01a6e38d7f3
Author: Kenneth Knowles 
Date:   2016-08-04T02:56:33Z

Port mentions of OldDoFn in PipelineOptions

commit 866d2c7dda01e04ff040b2ed655e6390c6b56ef4
Author: Kenneth Knowles 
Date:   2016-08-04T03:15:12Z

Port easy Java SDK tests to new DoFn

commit 2504240de6115139addf051c354fae4b3c49b67c
Author: Kenneth Knowles 
Date:   2016-08-04T03:15:58Z

Port PAssert to new DoFn

commit d16cc7f7dc14eaab1564b42177980ada149f7f99
Author: Kenneth Knowles 
Date:   2016-08-04T03:22:26Z

Port easy I/O transforms to new DoFn

commit 3f949838df812012175a086c287e99c25bca894e
Author: Kenneth Knowles 
Date:   2016-08-04T03:27:28Z

Port easy transforms to new DoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407011#comment-15407011
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/781

[BEAM-498] Port examples to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers pretty nice to be showcasing the `BoundedWindow` parameter 
in a couple of them.

Any other reviewer can feel free to LGTM if they are happy with it. It is 
pretty mechanical.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-new-DoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/781.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #781


commit c31986f5ee31c34538a73057638a45efd796caf5
Author: Kenneth Knowles 
Date:   2016-08-04T01:54:22Z

Port example tests to new DoFn

commit 8e6c0eaf66d7e5e6f2319aa13efe627b9fe3002a
Author: Kenneth Knowles 
Date:   2016-08-04T02:01:16Z

Port TfIdf example to new DoFn

commit c3c42aafc1987d06c55e76e669f8e2c0b7b59b2c
Author: Kenneth Knowles 
Date:   2016-08-04T02:03:11Z

Port TopWikipediaSessions example to new DoFn

commit 380f26afc73620abecbd38118095b96f1a91f030
Author: Kenneth Knowles 
Date:   2016-08-04T02:04:50Z

Port GameState Java 8 example to new DoFn

commit 1584666cb2c5ef3d8c9edfefe4d8cd7b366ae133
Author: Kenneth Knowles 
Date:   2016-08-04T02:06:26Z

Port the UserScore example to new DoFn

commit 86e90246d582327330af7f3212b6ed2c6a4f6af7
Author: Kenneth Knowles 
Date:   2016-08-04T02:07:56Z

Port StreamingWordExtract example to new DoFn

commit 30da6afd8d06f63f0a0a7ebafc51e8d30217763c
Author: Kenneth Knowles 
Date:   2016-08-04T02:08:19Z

fixup! UserScore

commit 18ee240879a0edc738355746f13b5c6b967babf7
Author: Kenneth Knowles 
Date:   2016-08-04T02:09:39Z

Port TrafficMaxLaneFlow to new DoFn

commit 589337562cf59c511dbc49030a88110cbfcd5a3a
Author: Kenneth Knowles 
Date:   2016-08-04T02:10:43Z

Port TrafficeRoutes example to new DoFn

commit 616411a1263604182f4cab7e899de3df22fc734d
Author: Kenneth Knowles 
Date:   2016-08-04T02:12:08Z

Port DatastoreWordCount example to new DoFn

commit 4878f0b274c656f4d9951f471b7ef346fca58d1f
Author: Kenneth Knowles 
Date:   2016-08-04T02:13:19Z

Port BigQueryTornadoes example to new DoFn

commit 480926d9591bfa0dace4af6d6883650bae61bb99
Author: Kenneth Knowles 
Date:   2016-08-04T02:13:58Z

Port MaxPerKeyExamples to new DoFn

commit 607ed16fdc38ad19fc711844e0c55da6306d0882
Author: Kenneth Knowles 
Date:   2016-08-04T02:14:37Z

Port CombinePerKeyExamples to new DoFn

commit e2262521eb4e84a258bfff03edab1440e91fd9f3
Author: Kenneth Knowles 
Date:   2016-08-04T02:15:56Z

Port TriggerExample to new DoFn

commit 8b376606d9a956a2be9b70508010a19d34584d81
Author: Kenneth Knowles 
Date:   2016-08-04T02:17:26Z

Port JoinExamples to new DoFn

commit 00e19ae9e690e35e73a0f8aff2c1a371d80c
Author: Kenneth Knowles 
Date:   2016-08-04T02:18:07Z

Port FilterExamples to new DoFn

commit 16b9ca531970b6b32c91229df80926fa0d99714c
Author: Kenneth Knowles 
Date:   2016-08-04T02:18:38Z

fixup! TriggerExample

commit ba47f11fb0d1aa99141ab12a4a3665a52d1e016e
Author: Kenneth Knowles 
Date:   2016-08-04T02:19:38Z

Fix mention of DoFn in WordCountTest




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #781: [BEAM-498] Port examples to new DoFn

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/781

[BEAM-498] Port examples to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers pretty nice to be showcasing the `BoundedWindow` parameter 
in a couple of them.

Any other reviewer can feel free to LGTM if they are happy with it. It is 
pretty mechanical.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-new-DoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/781.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #781


commit c31986f5ee31c34538a73057638a45efd796caf5
Author: Kenneth Knowles 
Date:   2016-08-04T01:54:22Z

Port example tests to new DoFn

commit 8e6c0eaf66d7e5e6f2319aa13efe627b9fe3002a
Author: Kenneth Knowles 
Date:   2016-08-04T02:01:16Z

Port TfIdf example to new DoFn

commit c3c42aafc1987d06c55e76e669f8e2c0b7b59b2c
Author: Kenneth Knowles 
Date:   2016-08-04T02:03:11Z

Port TopWikipediaSessions example to new DoFn

commit 380f26afc73620abecbd38118095b96f1a91f030
Author: Kenneth Knowles 
Date:   2016-08-04T02:04:50Z

Port GameState Java 8 example to new DoFn

commit 1584666cb2c5ef3d8c9edfefe4d8cd7b366ae133
Author: Kenneth Knowles 
Date:   2016-08-04T02:06:26Z

Port the UserScore example to new DoFn

commit 86e90246d582327330af7f3212b6ed2c6a4f6af7
Author: Kenneth Knowles 
Date:   2016-08-04T02:07:56Z

Port StreamingWordExtract example to new DoFn

commit 30da6afd8d06f63f0a0a7ebafc51e8d30217763c
Author: Kenneth Knowles 
Date:   2016-08-04T02:08:19Z

fixup! UserScore

commit 18ee240879a0edc738355746f13b5c6b967babf7
Author: Kenneth Knowles 
Date:   2016-08-04T02:09:39Z

Port TrafficMaxLaneFlow to new DoFn

commit 589337562cf59c511dbc49030a88110cbfcd5a3a
Author: Kenneth Knowles 
Date:   2016-08-04T02:10:43Z

Port TrafficeRoutes example to new DoFn

commit 616411a1263604182f4cab7e899de3df22fc734d
Author: Kenneth Knowles 
Date:   2016-08-04T02:12:08Z

Port DatastoreWordCount example to new DoFn

commit 4878f0b274c656f4d9951f471b7ef346fca58d1f
Author: Kenneth Knowles 
Date:   2016-08-04T02:13:19Z

Port BigQueryTornadoes example to new DoFn

commit 480926d9591bfa0dace4af6d6883650bae61bb99
Author: Kenneth Knowles 
Date:   2016-08-04T02:13:58Z

Port MaxPerKeyExamples to new DoFn

commit 607ed16fdc38ad19fc711844e0c55da6306d0882
Author: Kenneth Knowles 
Date:   2016-08-04T02:14:37Z

Port CombinePerKeyExamples to new DoFn

commit e2262521eb4e84a258bfff03edab1440e91fd9f3
Author: Kenneth Knowles 
Date:   2016-08-04T02:15:56Z

Port TriggerExample to new DoFn

commit 8b376606d9a956a2be9b70508010a19d34584d81
Author: Kenneth Knowles 
Date:   2016-08-04T02:17:26Z

Port JoinExamples to new DoFn

commit 00e19ae9e690e35e73a0f8aff2c1a371d80c
Author: Kenneth Knowles 
Date:   2016-08-04T02:18:07Z

Port FilterExamples to new DoFn

commit 16b9ca531970b6b32c91229df80926fa0d99714c
Author: Kenneth Knowles 
Date:   2016-08-04T02:18:38Z

fixup! TriggerExample

commit ba47f11fb0d1aa99141ab12a4a3665a52d1e016e
Author: Kenneth Knowles 
Date:   2016-08-04T02:19:38Z

Fix mention of DoFn in WordCountTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-532) Let users specify compression mode in WordCount

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-532:


 Summary: Let users specify compression mode in WordCount
 Key: BEAM-532
 URL: https://issues.apache.org/jira/browse/BEAM-532
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin
Priority: Minor


Users often ask about compression. Let's build this into the default example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #780: WordCount: add the option for users to spe...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/780

WordCount: add the option for users to specify compression

Also copy to archetypes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam word-count-compression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/780.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #780


commit 6e540a0ed78a504858123cd68e4cd257f9505144
Author: Dan Halperin 
Date:   2016-08-04T01:47:00Z

WordCount: add the option for users to specify compression

Also copy to archetypes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-531) Add support for getting aggregated values with dataflow runner

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-531:


 Summary: Add support for getting aggregated values with dataflow 
runner
 Key: BEAM-531
 URL: https://issues.apache.org/jira/browse/BEAM-531
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Charles Chen
Priority: Minor


The SDK for Python cannot extract metrics from the Dataflow service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-530) Decide where to place the tests and examples

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-530:


 Summary: Decide where to place the tests and examples
 Key: BEAM-530
 URL: https://issues.apache.org/jira/browse/BEAM-530
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor


Right now they are literally part of the package space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-529) Check immutability violations in DirectPipelineRunner

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-529:


 Summary: Check immutability violations in DirectPipelineRunner
 Key: BEAM-529
 URL: https://issues.apache.org/jira/browse/BEAM-529
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor


Users are going to mutate inputs and outputs of DoFn inappropriately. We should 
help their tests fail to catch such mistakes. (Similar to the 
DirectPipelineRunner in Java SDK)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[incubator-beam] Git Push Summary

[kenn]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150210 [deleted] 0f05e56cb
  refs/tags/v0.3.20150211 [deleted] 5a11734d6
  refs/tags/v0.3.20150227 [deleted] 0aa4d9d53
  refs/tags/v0.3.20150326 [deleted] 8192a7ef6
  refs/tags/v0.4.20150414 [deleted] 843eec882
  refs/tags/v0.4.20150602 [deleted] bdde37492
  refs/tags/v0.4.20150710 [deleted] ffb182ee3
  refs/tags/v0.4.20150727 [deleted] 39ba98739
  refs/tags/v1.0.0 [deleted] f971e4be4
  refs/tags/v1.1.0 [deleted] 469dd0b83
  refs/tags/v1.2.0 [deleted] 8e1de2021
  refs/tags/v1.2.1 [deleted] eac7a85b0
  refs/tags/v1.3.0 [deleted] 0c637d38a
  refs/tags/v1.4.0 [deleted] 02acad47e
  refs/tags/v1.5.0 [deleted] 05755bb18
  refs/tags/v1.5.1 [deleted] beb506c32
  refs/tags/v1.6.0 [deleted] 79e9b3164

[incubator-beam] Git Push Summary

[kenn]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20141216 [deleted] 71b4fe2ff
  refs/tags/v0.3.20150109 [deleted] adadb6103

[jira] [Created] (BEAM-528) Add @experimental annotations

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-528:


 Summary: Add @experimental annotations 
 Key: BEAM-528
 URL: https://issues.apache.org/jira/browse/BEAM-528
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


Experimental/deprecation warnings: use the warnings standard module in 
conjunction with decorators as described here:

https://docs.python.org/2/library/warnings.html

Some code sample for a deprecated decorator that is kinda/sorta similar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[incubator-beam] Git Push Summary

[kenn]

Repository: incubator-beam
Updated Tags:  refs/tags/apache-dataflow-ancestor [deleted] d4dcaaa05

[GitHub] incubator-beam pull request #779: [BEAM-522] Fixes GcsIO.exists() to properl...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/779

[BEAM-522] Fixes GcsIO.exists() to properly handle files that do not exist

Currently this invocation fails for non existing files instead of returning 
false.

Updates FileSink.finalize_write() so that we capture and log any transient 
errors that get thrown at the channel_factory.exists() call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
sink_finalize_fix_idempotency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/779.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #779


commit 792c3b5c79b6e979bc34bcf457f8a33cebd74daf
Author: Chamikara Jayalath 
Date:   2016-08-04T01:25:41Z

Fixes GcsIO.exists() to properly handle files that do not exist.

Currently this invocation fails for non existing files instead of returning 
false.

Updates FileSink.finalize_write() so that we capture and log any transient 
errors that get thrown at the channel_factory.exists() call.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-522) Update FileSink.finalize_write() to be idempotent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406959#comment-15406959
 ] 

ASF GitHub Bot commented on BEAM-522:
-

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/779

[BEAM-522] Fixes GcsIO.exists() to properly handle files that do not exist

Currently this invocation fails for non existing files instead of returning 
false.

Updates FileSink.finalize_write() so that we capture and log any transient 
errors that get thrown at the channel_factory.exists() call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
sink_finalize_fix_idempotency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/779.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #779


commit 792c3b5c79b6e979bc34bcf457f8a33cebd74daf
Author: Chamikara Jayalath 
Date:   2016-08-04T01:25:41Z

Fixes GcsIO.exists() to properly handle files that do not exist.

Currently this invocation fails for non existing files instead of returning 
false.

Updates FileSink.finalize_write() so that we capture and log any transient 
errors that get thrown at the channel_factory.exists() call.




> Update FileSink.finalize_write() to be idempotent
> -
>
> Key: BEAM-522
> URL: https://issues.apache.org/jira/browse/BEAM-522
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently FileSink.finelize_write() in fileio.py [1] performs following 
> operations.
> (1) Obtains a list of temporary files as a side input
> (2) Renames each temporary file to the location where final output should be 
> stored.
> iobase.Sink.finalize_write() operation should be idempotent since runner 
> implementations may call this operation multiple times due to task failures. 
> Current implementation is not idempotent because if we re-run the operation 
> after renaming a sub-set of files, the operations may fail due to not being 
> able to find some files at source location (for example, [2] for GCS files).
> We can fix this by checking if the destination file is already available 
> before performing the rename and not performing the rename for files that are 
> already available at the destination.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L503
> [2] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/gcsio.py#L187
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[incubator-beam] Git Push Summary

[kenn]

Repository: incubator-beam
Updated Tags:  refs/tags/OldDoFn [created] 388816a80
  refs/tags/apache-dataflow-ancestor [created] d4dcaaa05
  refs/tags/v0.3.20141216 [created] 71b4fe2ff
  refs/tags/v0.3.20150109 [created] adadb6103
  refs/tags/v0.3.20150210 [created] 0f05e56cb
  refs/tags/v0.3.20150211 [created] 5a11734d6
  refs/tags/v0.3.20150227 [created] 0aa4d9d53
  refs/tags/v0.3.20150326 [created] 8192a7ef6
  refs/tags/v0.4.20150414 [created] 843eec882
  refs/tags/v0.4.20150602 [created] bdde37492
  refs/tags/v0.4.20150710 [created] ffb182ee3
  refs/tags/v0.4.20150727 [created] 39ba98739
  refs/tags/v1.0.0 [created] f971e4be4
  refs/tags/v1.1.0 [created] 469dd0b83
  refs/tags/v1.2.0 [created] 8e1de2021
  refs/tags/v1.2.1 [created] eac7a85b0
  refs/tags/v1.3.0 [created] 0c637d38a
  refs/tags/v1.4.0 [created] 02acad47e
  refs/tags/v1.5.0 [created] 05755bb18
  refs/tags/v1.5.1 [created] beb506c32
  refs/tags/v1.6.0 [created] 79e9b3164

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406954#comment-15406954
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/758


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[18/19] incubator-beam git commit: Port DebuggingWordCount example from OldDoFn to DoFn

[dhalperi]

Port DebuggingWordCount example from OldDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/49d2f170
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/49d2f170
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/49d2f170

Branch: refs/heads/master
Commit: 49d2f1706f69c5106a9082ffd2fecaf69b2d868c
Parents: ca9e337
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:29:18 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../java/org/apache/beam/examples/DebuggingWordCount.java| 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/49d2f170/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
index 3c43152..c1b273c 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
@@ -24,7 +24,7 @@ import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Aggregator;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.values.KV;
@@ -106,8 +106,8 @@ import java.util.regex.Pattern;
  * overridden with {@code --inputFile}.
  */
 public class DebuggingWordCount {
-  /** A OldDoFn that filters for a specific key based upon a regular 
expression. */
-  public static class FilterTextFn extends OldDoFn, 
KV> {
+  /** A DoFn that filters for a specific key based upon a regular expression. 
*/
+  public static class FilterTextFn extends DoFn, KV> {
 /**
  * Concept #1: The logger below uses the fully qualified class name of 
FilterTextFn
  * as the logger. All log statements emitted by this logger will be 
referenced by this name
@@ -133,7 +133,7 @@ public class DebuggingWordCount {
 private final Aggregator unmatchedWords =
 createAggregator("umatchedWords", new Sum.SumLongFn());
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   if (filter.matcher(c.element().getKey()).matches()) {
 // Log at the "DEBUG" level each element that we match. When executing 
this pipeline

[05/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
index cafe873..517f968 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
@@ -24,6 +24,7 @@ import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasName
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasType;
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasValue;
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.includesDisplayDataFrom;
+
 import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.empty;
 import static org.hamcrest.Matchers.everyItem;
@@ -40,7 +41,7 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.fail;
 
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
@@ -54,7 +55,6 @@ import com.google.common.testing.EqualsTester;
 
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
-
 import org.hamcrest.CustomTypeSafeMatcher;
 import org.hamcrest.FeatureMatcher;
 import org.hamcrest.Matcher;
@@ -1053,7 +1053,7 @@ public class DisplayDataTest implements Serializable {
   private static class IdentityTransform extends PTransform {
 @Override
 public PCollection apply(PCollection input) {
-  return input.apply(ParDo.of(new DoFn() {
+  return input.apply(ParDo.of(new OldDoFn() {
 @Override
 public void processElement(ProcessContext c) throws Exception {
   c.output(c.element());

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
index 10a2a7e..97667a3 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
@@ -29,9 +29,9 @@ import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.RunnableOnService;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFn.RequiresWindowAccess;
 import org.apache.beam.sdk.transforms.DoFnTester;
+import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.OldDoFn.RequiresWindowAccess;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -85,8 +85,8 @@ public class CoGroupByKeyTest implements Serializable {
   .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), 
StringUtf8Coder.of(;
 }
 return input
-.apply("Identity" + name, ParDo.of(new DoFn,
- KV>() {
+.apply("Identity" + name, ParDo.of(new OldDoFn,
+ KV>() {
   @Override
   public void processElement(ProcessContext c) {
 c.output(c.element());
@@ -313,11 +313,11 @@ public class CoGroupByKeyTest implements Serializable {
   }
 
   /**
-   * A DoFn used in testCoGroupByKeyWithWindowing(), to test processing the
+   * A OldDoFn used in testCoGroupByKeyWithWindowing(), to test processing the
* results of a CoGroupByKey.
*/
   private static class ClickOfPurchaseFn extends
-  DoFn, KV> implements 
RequiresWindowAccess {
+  OldDoFn, KV> implements 
RequiresWindowAccess {
 private final TupleTag clicksTag;
 
 private final TupleTag purchasesTag;
@@ -347,11 +347,11 @@ public class CoGroupByKeyTest implements Serializable {
 
 
   /**
-   * A DoFn used in testCoGroupByKeyHandleResults(), to test processing the

[17/19] incubator-beam git commit: Update Dataflow runner to worker container supporting OldDoFn

[dhalperi]

Update Dataflow runner to worker container supporting OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3466a0e7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3466a0e7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3466a0e7

Branch: refs/heads/master
Commit: 3466a0e72d1f025c2d4562635a20730778b497b0
Parents: e07c339
Author: Kenneth Knowles 
Authored: Wed Aug 3 17:47:23 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3466a0e7/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index d762d50..abcf415 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -212,9 +212,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160714";
+  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160804-dofn";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160714";
+  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160804-dofn";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[13/19] incubator-beam git commit: Port microbenchmarks to new vocabulary

[dhalperi]

Port microbenchmarks to new vocabulary


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e07c3397
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e07c3397
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e07c3397

Branch: refs/heads/master
Commit: e07c3397d268f50cc879362227a6887cc52f4a3b
Parents: 3236eec
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:29:51 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../transforms/DoFnReflectorBenchmark.java  | 27 ++--
 1 file changed, 14 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e07c3397/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/transforms/DoFnReflectorBenchmark.java
--
diff --git 
a/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/transforms/DoFnReflectorBenchmark.java
 
b/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/transforms/DoFnReflectorBenchmark.java
index fd75e95..233b8be 100644
--- 
a/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/transforms/DoFnReflectorBenchmark.java
+++ 
b/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/transforms/DoFnReflectorBenchmark.java
@@ -53,9 +53,10 @@ public class DoFnReflectorBenchmark {
   private OldDoFn oldDoFn = new UpperCaseOldDoFn();
   private DoFn doFn = new UpperCaseDoFn();
 
-  private StubDoFnProcessContext stubDoFnContext = new 
StubDoFnProcessContext(oldDoFn, ELEMENT);
-  private StubDoFnWithContextProcessContext stubDoFnWithContextContext =
-  new StubDoFnWithContextProcessContext(doFn, ELEMENT);
+  private StubOldDoFnProcessContext stubOldDoFnContext = new 
StubOldDoFnProcessContext(oldDoFn,
+  ELEMENT);
+  private StubDoFnProcessContext stubDoFnContext =
+  new StubDoFnProcessContext(doFn, ELEMENT);
   private ExtraContextFactory extraContextFactory =
   new ExtraContextFactory() {
 
@@ -83,21 +84,21 @@ public class DoFnReflectorBenchmark {
   }
 
   @Benchmark
-  public String invokeDoFn() throws Exception {
-oldDoFn.processElement(stubDoFnContext);
+  public String invokeOldDoFn() throws Exception {
+oldDoFn.processElement(stubOldDoFnContext);
 return stubDoFnContext.output;
   }
 
   @Benchmark
   public String invokeDoFnWithContextViaAdaptor() throws Exception {
-adaptedDoFnWithContext.processElement(stubDoFnContext);
-return stubDoFnContext.output;
+adaptedDoFnWithContext.processElement(stubOldDoFnContext);
+return stubOldDoFnContext.output;
   }
 
   @Benchmark
   public String invokeDoFnWithContext() throws Exception {
-invoker.invokeProcessElement(stubDoFnWithContextContext, 
extraContextFactory);
-return stubDoFnWithContextContext.output;
+invoker.invokeProcessElement(stubDoFnContext, extraContextFactory);
+return stubDoFnContext.output;
   }
 
   private static class UpperCaseOldDoFn extends OldDoFn {
@@ -116,12 +117,12 @@ public class DoFnReflectorBenchmark {
 }
   }
 
-  private static class StubDoFnProcessContext extends OldDoFn.ProcessContext {
+  private static class StubOldDoFnProcessContext extends OldDoFn.ProcessContext {
 
 private final String element;
 private String output;
 
-public StubDoFnProcessContext(OldDoFn fn, String element) {
+public StubOldDoFnProcessContext(OldDoFn fn, String 
element) {
   fn.super();
   this.element = element;
 }
@@ -186,12 +187,12 @@ public class DoFnReflectorBenchmark {
 }
   }
 
-  private static class StubDoFnWithContextProcessContext
+  private static class StubDoFnProcessContext
   extends DoFn.ProcessContext {
 private final String element;
 private  String output;
 
-public StubDoFnWithContextProcessContext(DoFn fn, String 
element) {
+public StubDoFnProcessContext(DoFn fn, String element) {
   fn.super();
   this.element = element;
 }

[02/19] incubator-beam git commit: Rename DoFnWithContext to DoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3bcb6f46/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 0a6eab0..7fe053c 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -168,17 +168,17 @@ public class ParDoTest implements Serializable {
 public void processElement(OldDoFn.ProcessContext c) 
throws Exception {}
   }
 
-  static class TestDoFnWithContext extends DoFnWithContext {
+  static class TestDoFn extends DoFn {
 enum State { UNSTARTED, STARTED, PROCESSING, FINISHED }
 State state = State.UNSTARTED;
 
 final List sideInputViews = new ArrayList<>();
 final List sideOutputTupleTags = new ArrayList<>();
 
-public TestDoFnWithContext() {
+public TestDoFn() {
 }
 
-public TestDoFnWithContext(List sideInputViews,
+public TestDoFn(List sideInputViews,
 List sideOutputTupleTags) {
   this.sideInputViews.addAll(sideInputViews);
   this.sideOutputTupleTags.addAll(sideOutputTupleTags);
@@ -362,7 +362,7 @@ public class ParDoTest implements Serializable {
 
 PCollection output = pipeline
 .apply(Create.of(inputs))
-.apply(ParDo.of(new TestDoFnWithContext()));
+.apply(ParDo.of(new TestDoFn()));
 
 PAssert.that(output)
 .satisfies(ParDoTest.HasExpectedOutput.forInput(inputs));
@@ -1426,7 +1426,7 @@ public class ParDoTest implements Serializable {
 
   @Test
   public void testDoFnWithContextDisplayData() {
-DoFnWithContext fn = new DoFnWithContext() 
{
+DoFn fn = new DoFn() {
   @ProcessElement
   public void proccessElement(ProcessContext c) {}
 
@@ -1445,7 +1445,7 @@ public class ParDoTest implements Serializable {
 
   @Test
   public void testWithOutputTagsDisplayData() {
-DoFnWithContext fn = new DoFnWithContext() 
{
+DoFn fn = new DoFn() {
   @ProcessElement
   public void proccessElement(ProcessContext c) {}
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3bcb6f46/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/dofnreflector/DoFnReflectorTestHelper.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/dofnreflector/DoFnReflectorTestHelper.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/dofnreflector/DoFnReflectorTestHelper.java
index 5ff2bf1..90fba12 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/dofnreflector/DoFnReflectorTestHelper.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/dofnreflector/DoFnReflectorTestHelper.java
@@ -17,8 +17,8 @@
  */
 package org.apache.beam.sdk.transforms.dofnreflector;
 
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFnReflectorTest.Invocations;
-import org.apache.beam.sdk.transforms.DoFnWithContext;
 
 /**
  * Test helper for DoFnReflectorTest, which needs to test package-private 
access
@@ -26,7 +26,7 @@ import org.apache.beam.sdk.transforms.DoFnWithContext;
  */
 public class DoFnReflectorTestHelper {
 
-  private static class StaticPrivateDoFn extends DoFnWithContext {
+  private static class StaticPrivateDoFn extends DoFn {
 final Invocations invocations;
 
 public StaticPrivateDoFn(Invocations invocations) {
@@ -39,7 +39,7 @@ public class DoFnReflectorTestHelper {
 }
   }
 
-  private class InnerPrivateDoFn extends DoFnWithContext {
+  private class InnerPrivateDoFn extends DoFn {
 final Invocations invocations;
 
 public InnerPrivateDoFn(Invocations invocations) {
@@ -52,7 +52,7 @@ public class DoFnReflectorTestHelper {
 }
   }
 
-  static class StaticPackagePrivateDoFn extends DoFnWithContext {
+  static class StaticPackagePrivateDoFn extends DoFn {
 final Invocations invocations;
 
 public StaticPackagePrivateDoFn(Invocations invocations) {
@@ -65,7 +65,7 @@ public class DoFnReflectorTestHelper {
 }
   }
 
-  class InnerPackagePrivateDoFn extends DoFnWithContext {
+  class InnerPackagePrivateDoFn extends DoFn {
 final Invocations invocations;
 
 public InnerPackagePrivateDoFn(Invocations invocations) {
@@ -78,25 +78,25 @@ public class DoFnReflectorTestHelper {
 }
   }
 
-  public 

[12/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

Rename DoFn to OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a64baf48
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a64baf48
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a64baf48

Branch: refs/heads/master
Commit: a64baf4878f28e98da696dacc587c1151d0cdb9e
Parents: 388816a
Author: Kenneth Knowles 
Authored: Fri Jul 22 13:00:10 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:52 2016 -0700

--
 .../beam/examples/DebuggingWordCount.java   |   6 +-
 .../apache/beam/examples/MinimalWordCount.java  |   7 +-
 .../apache/beam/examples/WindowedWordCount.java |  10 +-
 .../org/apache/beam/examples/WordCount.java |   8 +-
 .../examples/common/PubsubFileInjector.java |   6 +-
 .../beam/examples/complete/AutoComplete.java|  16 +-
 .../examples/complete/StreamingWordExtract.java |  12 +-
 .../apache/beam/examples/complete/TfIdf.java|  16 +-
 .../examples/complete/TopWikipediaSessions.java |  12 +-
 .../examples/complete/TrafficMaxLaneFlow.java   |  10 +-
 .../beam/examples/complete/TrafficRoutes.java   |  12 +-
 .../examples/cookbook/BigQueryTornadoes.java|   6 +-
 .../cookbook/CombinePerKeyExamples.java |   6 +-
 .../examples/cookbook/DatastoreWordCount.java   |  11 +-
 .../beam/examples/cookbook/FilterExamples.java  |  12 +-
 .../beam/examples/cookbook/JoinExamples.java|  10 +-
 .../examples/cookbook/MaxPerKeyExamples.java|   6 +-
 .../beam/examples/cookbook/TriggerExample.java  |  12 +-
 .../org/apache/beam/examples/WordCountTest.java |   2 +-
 .../examples/complete/AutoCompleteTest.java |   4 +-
 .../examples/cookbook/TriggerExampleTest.java   |   4 +-
 .../beam/examples/complete/game/GameStats.java  |  10 +-
 .../beam/examples/complete/game/UserScore.java  |   4 +-
 .../complete/game/utils/WriteToBigQuery.java|  12 +-
 .../game/utils/WriteWindowedToBigQuery.java |   8 +-
 .../examples/complete/game/UserScoreTest.java   |   2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  12 +-
 .../core/UnboundedReadFromBoundedSource.java|   2 +-
 .../apache/beam/sdk/util/AssignWindowsDoFn.java |  10 +-
 .../org/apache/beam/sdk/util/DoFnRunner.java|  21 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  54 +-
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  24 +-
 .../beam/sdk/util/GroupAlsoByWindowsDoFn.java   |   6 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   4 +-
 .../sdk/util/GroupByKeyViaGroupByKeyOnly.java   |   6 +-
 .../sdk/util/LateDataDroppingDoFnRunner.java|   4 +-
 .../apache/beam/sdk/util/PaneInfoTracker.java   |   1 -
 .../apache/beam/sdk/util/ReduceFnRunner.java|   4 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |  12 +-
 .../org/apache/beam/sdk/util/WatermarkHold.java |   1 -
 .../beam/sdk/util/ReduceFnRunnerTest.java   |   1 +
 .../apache/beam/sdk/util/ReduceFnTester.java|   1 +
 .../beam/sdk/util/SimpleDoFnRunnerTest.java |   6 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   4 +-
 .../ImmutabilityCheckingBundleFactory.java  |   4 +-
 .../beam/runners/direct/ParDoEvaluator.java |   4 +-
 .../direct/ParDoMultiEvaluatorFactory.java  |  11 +-
 .../direct/ParDoSingleEvaluatorFactory.java |  11 +-
 .../direct/TransformEvaluatorFactory.java   |   6 +-
 .../direct/WriteWithShardingFactory.java|   4 +-
 .../ConsumerTrackingPipelineVisitorTest.java|  22 +-
 .../beam/runners/direct/DirectRunnerTest.java   |  24 +-
 .../ImmutabilityCheckingBundleFactoryTest.java  |   6 +-
 .../ImmutabilityEnforcementFactoryTest.java |   6 +-
 .../direct/KeyedPValueTrackingVisitorTest.java  |   6 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  |  10 +-
 .../direct/ParDoSingleEvaluatorFactoryTest.java |  10 +-
 .../runners/direct/WatermarkManagerTest.java|   7 +-
 .../beam/runners/flink/examples/TFIDF.java  |  16 +-
 .../beam/runners/flink/examples/WordCount.java  |   4 +-
 .../flink/examples/streaming/AutoComplete.java  |  16 +-
 .../flink/examples/streaming/JoinExamples.java  |   8 +-
 .../examples/streaming/KafkaIOExamples.java |   4 +-
 .../KafkaWindowedWordCountExample.java  |   6 +-
 .../examples/streaming/WindowedWordCount.java   |   6 +-
 .../FlinkBatchTransformTranslators.java |  12 +-
 .../FlinkStreamingTransformTranslators.java |   9 +-
 .../functions/FlinkDoFnFunction.java|  10 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   8 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  10 +-
 .../FlinkMultiOutputProcessContext.java |   6 +-
 .../functions/FlinkNoElementAssignContext.java  |   8 +-
 .../functions/FlinkPartialReduceFunction.java   |   8 +-
 .../functions/FlinkProcessContext.java  |  16 

[10/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
index 89243a3..a4af1b0 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
@@ -19,7 +19,7 @@ package org.apache.beam.runners.flink.translation.functions;
 
 import 
org.apache.beam.runners.flink.translation.utils.SerializedPipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollectionView;
@@ -30,13 +30,13 @@ import org.apache.flink.util.Collector;
 import java.util.Map;
 
 /**
- * Encapsulates a {@link org.apache.beam.sdk.transforms.DoFn}
+ * Encapsulates a {@link OldDoFn}
  * inside a Flink {@link 
org.apache.flink.api.common.functions.RichMapPartitionFunction}.
  */
 public class FlinkDoFnFunction
 extends RichMapPartitionFunction {
 
-  private final DoFn doFn;
+  private final OldDoFn doFn;
   private final SerializedPipelineOptions serializedOptions;
 
   private final Map sideInputs;
@@ -47,7 +47,7 @@ public class FlinkDoFnFunction
   private final WindowingStrategy windowingStrategy;
 
   public FlinkDoFnFunction(
-  DoFn doFn,
+  OldDoFn doFn,
   WindowingStrategy windowingStrategy,
   Map sideInputs,
   PipelineOptions options) {
@@ -56,7 +56,7 @@ public class FlinkDoFnFunction
 this.serializedOptions = new SerializedPipelineOptions(options);
 this.windowingStrategy = windowingStrategy;
 
-this.requiresWindowAccess = doFn instanceof DoFn.RequiresWindowAccess;
+this.requiresWindowAccess = doFn instanceof OldDoFn.RequiresWindowAccess;
 this.hasSideInputs = !sideInputs.isEmpty();
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
index 9074d72..2d36043 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
@@ -20,7 +20,7 @@ package org.apache.beam.runners.flink.translation.functions;
 import 
org.apache.beam.runners.flink.translation.utils.SerializedPipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.CombineFnBase;
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
@@ -60,7 +60,7 @@ public class FlinkMergingNonShuffleReduceFunction<
 
   private final CombineFnBase.PerKeyCombineFn 
combineFn;
 
-  private final DoFn, KV> doFn;
+  private final OldDoFn, KV> doFn;
 
   private final WindowingStrategy windowingStrategy;
 
@@ -81,8 +81,8 @@ public class FlinkMergingNonShuffleReduceFunction<
 
 this.serializedOptions = new SerializedPipelineOptions(pipelineOptions);
 
-// dummy DoFn because we need one for ProcessContext
-this.doFn = new DoFn, KV>() {
+// dummy OldDoFn because we need one for ProcessContext
+this.doFn = new OldDoFn, KV>() {
   @Override
   public void processElement(ProcessContext c) throws Exception {
 

[04/19] incubator-beam git commit: Rename NoOpDoFn to NoOpOldDoFn

[dhalperi]

Rename NoOpDoFn to NoOpOldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e1609664
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e1609664
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e1609664

Branch: refs/heads/master
Commit: e16096648fa92e72822dc02dd1958f5e53fd6fa0
Parents: a64baf4
Author: Kenneth Knowles 
Authored: Mon Jul 25 21:27:02 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:52 2016 -0700

--
 .../DoFnDelegatingAggregatorTest.java   |  10 +-
 .../sdk/transforms/DoFnWithContextTest.java |   2 +-
 .../apache/beam/sdk/transforms/NoOpDoFn.java| 144 ---
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java | 144 +++
 .../beam/sdk/transforms/OldDoFnContextTest.java |   2 +-
 .../apache/beam/sdk/transforms/OldDoFnTest.java |  12 +-
 6 files changed, 157 insertions(+), 157 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1609664/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnDelegatingAggregatorTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnDelegatingAggregatorTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnDelegatingAggregatorTest.java
index 2488042..4e8d06c 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnDelegatingAggregatorTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnDelegatingAggregatorTest.java
@@ -54,7 +54,7 @@ public class DoFnDelegatingAggregatorTest {
 
   @Test
   public void testAddValueWithoutDelegateThrowsException() {
-OldDoFn doFn = new NoOpDoFn<>();
+OldDoFn doFn = new NoOpOldDoFn<>();
 
 String name = "agg";
 CombineFn combiner = mockCombineFn(Double.class);
@@ -74,7 +74,7 @@ public class DoFnDelegatingAggregatorTest {
 String name = "agg";
 CombineFn combiner = mockCombineFn(Long.class);
 
-OldDoFn doFn = new NoOpDoFn<>();
+OldDoFn doFn = new NoOpOldDoFn<>();
 
 DelegatingAggregator aggregator =
 (DelegatingAggregator) doFn.createAggregator(name, 
combiner);
@@ -91,7 +91,7 @@ public class DoFnDelegatingAggregatorTest {
 String name = "agg";
 CombineFn combiner = mockCombineFn(Double.class);
 
-OldDoFn doFn = new NoOpDoFn<>();
+OldDoFn doFn = new NoOpOldDoFn<>();
 
 DelegatingAggregator aggregator =
 (DelegatingAggregator) doFn.createAggregator(name, 
combiner);
@@ -114,7 +114,7 @@ public class DoFnDelegatingAggregatorTest {
 String name = "agg";
 CombineFn combiner = mockCombineFn(Double.class);
 
-OldDoFn doFn = new NoOpDoFn<>();
+OldDoFn doFn = new NoOpOldDoFn<>();
 
 DelegatingAggregator aggregator =
 (DelegatingAggregator) doFn.createAggregator(name, 
combiner);
@@ -127,7 +127,7 @@ public class DoFnDelegatingAggregatorTest {
 String name = "agg";
 CombineFn combiner = mockCombineFn(Double.class);
 
-OldDoFn doFn = new NoOpDoFn<>();
+OldDoFn doFn = new NoOpOldDoFn<>();
 
 DelegatingAggregator aggregator =
 (DelegatingAggregator) doFn.createAggregator(name, 
combiner);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1609664/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
index 8b00c03..0a910b8 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
@@ -141,7 +141,7 @@ public class DoFnWithContextTest implements Serializable {
 
   @Test
   public void testDoFnWithContextUsingAggregators() {
-NoOpDoFn noOpFn = new NoOpDoFn<>();
+NoOpOldDoFn noOpFn = new NoOpOldDoFn<>();
 OldDoFn.Context context = noOpFn.context();
 
 OldDoFn fn = spy(noOpFn);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1609664/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpDoFn.java

[03/19] incubator-beam git commit: Rename DoFnWithContext to DoFn

[dhalperi]

Rename DoFnWithContext to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3bcb6f46
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3bcb6f46
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3bcb6f46

Branch: refs/heads/master
Commit: 3bcb6f46ad0ae483d1d8785edc2d9d5846c71a73
Parents: e160966
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:10:01 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:52 2016 -0700

--
 .../org/apache/beam/sdk/transforms/DoFn.java| 429 +++
 .../beam/sdk/transforms/DoFnReflector.java  |  84 ++--
 .../apache/beam/sdk/transforms/DoFnTester.java  |   2 +-
 .../beam/sdk/transforms/DoFnWithContext.java| 429 ---
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  16 +-
 .../beam/sdk/transforms/DoFnReflectorTest.java  |  86 ++--
 .../apache/beam/sdk/transforms/DoFnTest.java| 237 ++
 .../sdk/transforms/DoFnWithContextTest.java | 237 --
 .../apache/beam/sdk/transforms/ParDoTest.java   |  12 +-
 .../dofnreflector/DoFnReflectorTestHelper.java  |  26 +-
 .../transforms/DoFnReflectorBenchmark.java  |  30 +-
 12 files changed, 795 insertions(+), 795 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3bcb6f46/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
new file mode 100644
index 000..eb6753c
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.apache.beam.sdk.transforms.OldDoFn.DelegatingAggregator;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.HasDisplayData;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.WindowingInternals;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+import java.io.Serializable;
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The argument to {@link ParDo} providing the code to use to process
+ * elements of the input
+ * {@link org.apache.beam.sdk.values.PCollection}.
+ *
+ * See {@link ParDo} for more explanation, examples of use, and
+ * discussion of constraints on {@code DoFn}s, including their
+ * serializability, lack of access to global shared mutable state,
+ * requirements for failure tolerance, and benefits of optimization.
+ *
+ * {@code DoFn}s can be tested in a particular
+ * {@code Pipeline} by running that {@code Pipeline} on sample input
+ * and then checking its output.  Unit testing of a {@code DoFn},
+ * separately from any {@code ParDo} transform or {@code Pipeline},
+ * can be done via the {@link DoFnTester} harness.
+ *
+ * Implementations must define a method annotated with {@link 
ProcessElement}
+ * that satisfies 

[08/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnWithContext.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnWithContext.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnWithContext.java
index 8a83e44..b27163a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnWithContext.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnWithContext.java
@@ -24,7 +24,7 @@ import static com.google.common.base.Preconditions.checkState;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.DoFn.DelegatingAggregator;
+import org.apache.beam.sdk.transforms.OldDoFn.DelegatingAggregator;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -127,7 +127,7 @@ public abstract class DoFnWithContext 
implements Serializable,
  *
  * If invoked from {@link ProcessElement}), the timestamp
  * must not be older than the input element's timestamp minus
- * {@link DoFn#getAllowedTimestampSkew}.  The output element will
+ * {@link OldDoFn#getAllowedTimestampSkew}.  The output element will
  * be in the same windows as the input element.
  *
  * If invoked from {@link StartBundle} or {@link FinishBundle},
@@ -176,7 +176,7 @@ public abstract class DoFnWithContext 
implements Serializable,
  *
  * If invoked from {@link ProcessElement}), the timestamp
  * must not be older than the input element's timestamp minus
- * {@link DoFn#getAllowedTimestampSkew}.  The output element will
+ * {@link OldDoFn#getAllowedTimestampSkew}.  The output element will
  * be in the same windows as the input element.
  *
  * If invoked from {@link StartBundle} or {@link FinishBundle},
@@ -194,7 +194,7 @@ public abstract class DoFnWithContext 
implements Serializable,
   }
 
   /**
-   * Information accessible when running {@link DoFn#processElement}.
+   * Information accessible when running {@link OldDoFn#processElement}.
*/
   public abstract class ProcessContext extends Context {
 
@@ -358,13 +358,13 @@ public abstract class DoFnWithContext 
implements Serializable,
   /**
* Returns an {@link Aggregator} with aggregation logic specified by the
* {@link CombineFn} argument. The name provided must be unique across
-   * {@link Aggregator}s created within the DoFn. Aggregators can only be 
created
+   * {@link Aggregator}s created within the OldDoFn. Aggregators can only be 
created
* during pipeline construction.
*
* @param name the name of the aggregator
* @param combiner the {@link CombineFn} to use in the aggregator
* @return an aggregator for the provided name and combiner in the scope of
-   * this DoFn
+   * this OldDoFn
* @throws NullPointerException if the name or combiner is null
* @throws IllegalArgumentException if the given name collides with another
* aggregator in this scope
@@ -391,13 +391,13 @@ public abstract class DoFnWithContext 
implements Serializable,
   /**
* Returns an {@link Aggregator} with the aggregation logic specified by the
* {@link SerializableFunction} argument. The name provided must be unique
-   * across {@link Aggregator}s created within the DoFn. Aggregators can only 
be
+   * across {@link Aggregator}s created within the OldDoFn. Aggregators can 
only be
* created during pipeline construction.
*
* @param name the name of the aggregator
* @param combiner the {@link SerializableFunction} to use in the aggregator
* @return an aggregator for the provided name and combiner in the scope of
-   * this DoFn
+   * this OldDoFn
* @throws NullPointerException if the name or combiner is null
* @throws IllegalArgumentException if the given name collides with another
* aggregator in this scope

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
index a31799e..4466874 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
@@ -202,7 +202,7 @@ public class Filter extends PTransform

[14/19] incubator-beam git commit: Port MinimalWordCount example from OldDoFn to DoFn

[dhalperi]

Port MinimalWordCount example from OldDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4ceec0e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4ceec0e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4ceec0e8

Branch: refs/heads/master
Commit: 4ceec0e86f1c4e885168957299dbe81c61fbc7e7
Parents: 64481d0
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:28:42 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../java/org/apache/beam/examples/MinimalWordCount.java | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4ceec0e8/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
index ab0bb6d..df725e3 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
@@ -22,8 +22,8 @@ import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.values.KV;
@@ -89,12 +89,11 @@ public class MinimalWordCount {
 // the input text (a set of Shakespeare's texts).
 p.apply(TextIO.Read.from("gs://dataflow-samples/shakespeare/*"))
  // Concept #2: Apply a ParDo transform to our PCollection of text lines. 
This ParDo invokes a
- // OldDoFn (defined in-line) on each element that tokenizes the text line 
into individua
- // words.
+ // DoFn (defined in-line) on each element that tokenizes the text line 
into individual words.
  // The ParDo returns a PCollection, where each element is an 
individual word in
  // Shakespeare's collected texts.
- .apply("ExtractWords", ParDo.of(new OldDoFn() {
-   @Override
+ .apply("ExtractWords", ParDo.of(new DoFn() {
+   @ProcessElement
public void processElement(ProcessContext c) {
  for (String word : c.element().split("[^a-zA-Z']+")) {
if (!word.isEmpty()) {

[15/19] incubator-beam git commit: Port WindowedWordCount example from OldDoFn to DoFn

[dhalperi]

Port WindowedWordCount example from OldDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ca9e3372
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ca9e3372
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ca9e3372

Branch: refs/heads/master
Commit: ca9e337203208c7c5876f0710fb3a45430a5b3a8
Parents: 4ceec0e
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:29:01 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../org/apache/beam/examples/WindowedWordCount.java   | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ca9e3372/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
index 17f7da3..842cb54 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
@@ -27,7 +27,7 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
@@ -103,14 +103,14 @@ public class WindowedWordCount {
 static final int WINDOW_SIZE = 1;  // Default window duration in minutes
 
   /**
-   * Concept #2: A OldDoFn that sets the data element timestamp. This is a 
silly method, just for
+   * Concept #2: A DoFn that sets the data element timestamp. This is a silly 
method, just for
* this example, for the bounded data case.
*
* Imagine that many ghosts of Shakespeare are all typing madly at the 
same time to recreate
* his masterworks. Each line of the corpus will get a random associated 
timestamp somewhere in a
* 2-hour period.
*/
-  static class AddTimestampFn extends OldDoFn {
+  static class AddTimestampFn extends DoFn {
 private static final Duration RAND_RANGE = Duration.standardHours(2);
 private final Instant minTimestamp;
 
@@ -118,7 +118,7 @@ public class WindowedWordCount {
   this.minTimestamp = new Instant(System.currentTimeMillis());
 }
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   // Generate a timestamp that falls somewhere in the past two hours.
   long randMillis = (long) (Math.random() * RAND_RANGE.getMillis());
@@ -130,9 +130,9 @@ public class WindowedWordCount {
 }
   }
 
-  /** A OldDoFn that converts a Word and Count into a BigQuery table row. */
-  static class FormatAsTableRowFn extends OldDoFn, TableRow> {
-@Override
+  /** A DoFn that converts a Word and Count into a BigQuery table row. */
+  static class FormatAsTableRowFn extends DoFn, TableRow> {
+@ProcessElement
 public void processElement(ProcessContext c) {
   TableRow row = new TableRow()
   .set("word", c.element().getKey())

[16/19] incubator-beam git commit: Port AutoComplete example from OldDoFn to DoFn

[dhalperi]

Port AutoComplete example from OldDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3236eec2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3236eec2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3236eec2

Branch: refs/heads/master
Commit: 3236eec22a8902393e6becefb771b9a4768ccc50
Parents: 49d2f17
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:29:37 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../beam/examples/complete/AutoComplete.java| 30 ++--
 1 file changed, 15 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3236eec2/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
index 7b44af8..1ab39c9 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
@@ -36,9 +36,9 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.StreamingOptions;
 import org.apache.beam.sdk.options.Validation;
 import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Filter;
 import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.Partition;
@@ -130,8 +130,8 @@ public class AutoComplete {
 
 // Map the KV outputs of Count into our own CompletionCandiate class.
 .apply("CreateCompletionCandidates", ParDo.of(
-new OldDoFn, CompletionCandidate>() {
-  @Override
+new DoFn, CompletionCandidate>() {
+  @ProcessElement
   public void processElement(ProcessContext c) {
 c.output(new CompletionCandidate(c.element().getKey(), 
c.element().getValue()));
   }
@@ -209,8 +209,8 @@ public class AutoComplete {
 }
 
 private static class FlattenTops
-extends OldDoFn, 
CompletionCandidate> {
-  @Override
+extends DoFn, 
CompletionCandidate> {
+  @ProcessElement
   public void processElement(ProcessContext c) {
 for (CompletionCandidate cc : c.element().getValue()) {
   c.output(cc);
@@ -260,10 +260,10 @@ public class AutoComplete {
   }
 
   /**
-   * A OldDoFn that keys each candidate by all its prefixes.
+   * A DoFn that keys each candidate by all its prefixes.
*/
   private static class AllPrefixes
-  extends OldDoFn> {
+  extends DoFn> {
 private final int minPrefix;
 private final int maxPrefix;
 public AllPrefixes(int minPrefix) {
@@ -273,8 +273,8 @@ public class AutoComplete {
   this.minPrefix = minPrefix;
   this.maxPrefix = maxPrefix;
 }
-@Override
-  public void processElement(ProcessContext c) {
+@ProcessElement
+public void processElement(ProcessContext c) {
   String word = c.element().value;
   for (int i = minPrefix; i <= Math.min(word.length(), maxPrefix); i++) {
 c.output(KV.of(word.substring(0, i), c.element()));
@@ -341,8 +341,8 @@ public class AutoComplete {
   /**
* Takes as input a set of strings, and emits each #hashtag found therein.
*/
-  static class ExtractHashtags extends OldDoFn {
-@Override
+  static class ExtractHashtags extends DoFn {
+@ProcessElement
 public void processElement(ProcessContext c) {
   Matcher m = Pattern.compile("#\\S+").matcher(c.element());
   while (m.find()) {
@@ -351,8 +351,8 @@ public class AutoComplete {
 }
   }
 
-  static class FormatForBigquery extends OldDoFn, TableRow> {
-@Override
+  static class FormatForBigquery extends DoFn, TableRow> {
+@ProcessElement
 public void processElement(ProcessContext c) {
   List completions = new ArrayList<>();
   for (CompletionCandidate cc : c.element().getValue()) {
@@ -385,14 +385,14 @@ public class AutoComplete {
* Takes as input a the top candidates per prefix, and emits an entity
* suitable for writing to Datastore.
*/
-  static class 

[07/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
index 77c857c..7917aec 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
@@ -23,8 +23,8 @@ import static 
com.google.common.base.Preconditions.checkNotNull;
 import org.apache.beam.sdk.coders.AtomicCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.util.VarInt;
 
 import com.google.common.base.MoreObjects;
@@ -38,8 +38,8 @@ import java.util.Objects;
 /**
  * Provides information about the pane an element belongs to. Every pane is 
implicitly associated
  * with a window. Panes are observable only via the
- * {@link org.apache.beam.sdk.transforms.DoFn.ProcessContext#pane} method of 
the context
- * passed to a {@link DoFn#processElement} overridden method.
+ * {@link OldDoFn.ProcessContext#pane} method of the context
+ * passed to a {@link OldDoFn#processElement} overridden method.
  *
  * Note: This does not uniquely identify a pane, and should not be used for 
comparisons.
  */
@@ -74,8 +74,8 @@ public final class PaneInfo {
* definitions:
* 
* We'll call a pipeline 'simple' if it does not use
-   * {@link org.apache.beam.sdk.transforms.DoFn.Context#outputWithTimestamp} in
-   * any {@code DoFn}, and it uses the same
+   * {@link OldDoFn.Context#outputWithTimestamp} in
+   * any {@code OldDoFn}, and it uses the same
* {@link 
org.apache.beam.sdk.transforms.windowing.Window.Bound#withAllowedLateness}
* argument value on all windows (or uses the default of {@link 
org.joda.time.Duration#ZERO}).
* We'll call an element 'locally late', from the point of view of a 
computation on a

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index fe8b66f..03ff481 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -21,8 +21,8 @@ import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
-import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.display.DisplayData;
@@ -645,7 +645,7 @@ public class Window {
   // We first apply a (trivial) transform to the input PCollection to 
produce a new
   // PCollection. This ensures that we don't modify the windowing 
strategy of the input
   // which may be used elsewhere.
-  .apply("Identity", ParDo.of(new DoFn() {
+  .apply("Identity", ParDo.of(new OldDoFn() {
 @Override public void processElement(ProcessContext c) {
   c.output(c.element());
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
index a62444f..dd36367 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
@@ -107,7 +107,7 @@ public abstract class BaseExecutionContexthttp://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java 

[09/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SerializationTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SerializationTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SerializationTest.java
index 5b9eeff..5e96c46 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SerializationTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SerializationTest.java
@@ -30,7 +30,7 @@ import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.Sum;
@@ -145,9 +145,9 @@ public class SerializationTest {
   }
 
   /**
-   * A DoFn that tokenizes lines of text into individual words.
+   * A OldDoFn that tokenizes lines of text into individual words.
*/
-  static class ExtractWordsFn extends DoFn {
+  static class ExtractWordsFn extends OldDoFn {
 private static final Pattern WORD_BOUNDARY = 
Pattern.compile("[^a-zA-Z']+");
 private final Aggregator emptyLines =
 createAggregator("emptyLines", new Sum.SumLongFn());
@@ -173,9 +173,9 @@ public class SerializationTest {
   }
 
   /**
-   * A DoFn that converts a Word and Count into a printable string.
+   * A OldDoFn that converts a Word and Count into a printable string.
*/
-  private static class FormatCountsFn extends DoFn, 
StringHolder> {
+  private static class FormatCountsFn extends OldDoFn, 
StringHolder> {
 @Override
 public void processElement(ProcessContext c) {
   c.output(new StringHolder(c.element().getKey() + ": " + 
c.element().getValue()));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SideEffectsTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SideEffectsTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SideEffectsTest.java
index 60b7f71..5775565 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SideEffectsTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SideEffectsTest.java
@@ -28,7 +28,7 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.StringDelegateCoder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 
 import org.junit.After;
@@ -54,7 +54,7 @@ public class SideEffectsTest implements Serializable {
 
 pipeline.getCoderRegistry().registerCoder(URI.class, 
StringDelegateCoder.of(URI.class));
 
-pipeline.apply(Create.of("a")).apply(ParDo.of(new DoFn() {
+pipeline.apply(Create.of("a")).apply(ParDo.of(new OldDoFn() {
   @Override
   public void processElement(ProcessContext c) throws Exception {
 throw new UserException();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
index 904b448..c005f14 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
@@ -27,7 +27,7 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
@@ -122,7 +122,7 @@ public class KafkaStreamingTest {
 

[01/19] incubator-beam git commit: Port WordCount example from OldDoFn to DoFn

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 388816a80 -> 9a329aada


Port WordCount example from OldDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/64481d0c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/64481d0c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/64481d0c

Branch: refs/heads/master
Commit: 64481d0c2ed52a075ca1f0aa9946155aa9b13119
Parents: 3bcb6f4
Author: Kenneth Knowles 
Authored: Fri Jul 22 14:28:28 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:52 2016 -0700

--
 .../src/main/java/org/apache/beam/examples/WordCount.java | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/64481d0c/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
index 274d1ad..d3768a8 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
@@ -26,8 +26,8 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
@@ -97,14 +97,14 @@ public class WordCount {
 
   /**
* Concept #2: You can make your pipeline code less verbose by defining your 
DoFns statically out-
-   * of-line. This OldDoFn tokenizes lines of text into individual words; we 
pass it to a ParDo in
-   * the pipeline.
+   * of-line. This DoFn tokenizes lines of text into individual words; we pass 
it to a ParDo in the
+   * pipeline.
*/
-  static class ExtractWordsFn extends OldDoFn {
+  static class ExtractWordsFn extends DoFn {
 private final Aggregator emptyLines =
 createAggregator("emptyLines", new Sum.SumLongFn());
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   if (c.element().trim().isEmpty()) {
 emptyLines.addValue(1L);

[19/19] incubator-beam git commit: Closes #758

[dhalperi]

Closes #758


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9a329aad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9a329aad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9a329aad

Branch: refs/heads/master
Commit: 9a329aada82d3ca7f619b88eddea04bdd329d992
Parents: 388816a 3466a0e
Author: Dan Halperin 
Authored: Wed Aug 3 18:25:53 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 18:25:53 2016 -0700

--
 .../beam/examples/DebuggingWordCount.java   |   2 +-
 .../apache/beam/examples/MinimalWordCount.java  |   2 +-
 .../apache/beam/examples/WindowedWordCount.java |   4 +-
 .../org/apache/beam/examples/WordCount.java |   2 +-
 .../examples/common/PubsubFileInjector.java |   6 +-
 .../beam/examples/complete/AutoComplete.java|  14 +-
 .../examples/complete/StreamingWordExtract.java |  12 +-
 .../apache/beam/examples/complete/TfIdf.java|  16 +-
 .../examples/complete/TopWikipediaSessions.java |  12 +-
 .../examples/complete/TrafficMaxLaneFlow.java   |  10 +-
 .../beam/examples/complete/TrafficRoutes.java   |  12 +-
 .../examples/cookbook/BigQueryTornadoes.java|   6 +-
 .../cookbook/CombinePerKeyExamples.java |   6 +-
 .../examples/cookbook/DatastoreWordCount.java   |  11 +-
 .../beam/examples/cookbook/FilterExamples.java  |  12 +-
 .../beam/examples/cookbook/JoinExamples.java|  10 +-
 .../examples/cookbook/MaxPerKeyExamples.java|   6 +-
 .../beam/examples/cookbook/TriggerExample.java  |  12 +-
 .../org/apache/beam/examples/WordCountTest.java |   2 +-
 .../examples/complete/AutoCompleteTest.java |   4 +-
 .../examples/cookbook/TriggerExampleTest.java   |   4 +-
 .../beam/examples/complete/game/GameStats.java  |  10 +-
 .../beam/examples/complete/game/UserScore.java  |   4 +-
 .../complete/game/utils/WriteToBigQuery.java|  12 +-
 .../game/utils/WriteWindowedToBigQuery.java |   8 +-
 .../examples/complete/game/UserScoreTest.java   |   2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  12 +-
 .../core/UnboundedReadFromBoundedSource.java|   2 +-
 .../apache/beam/sdk/util/AssignWindowsDoFn.java |  10 +-
 .../org/apache/beam/sdk/util/DoFnRunner.java|  21 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  54 +-
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  24 +-
 .../beam/sdk/util/GroupAlsoByWindowsDoFn.java   |   6 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   4 +-
 .../sdk/util/GroupByKeyViaGroupByKeyOnly.java   |   6 +-
 .../sdk/util/LateDataDroppingDoFnRunner.java|   4 +-
 .../apache/beam/sdk/util/PaneInfoTracker.java   |   1 -
 .../apache/beam/sdk/util/ReduceFnRunner.java|   4 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |  12 +-
 .../org/apache/beam/sdk/util/WatermarkHold.java |   1 -
 .../beam/sdk/util/ReduceFnRunnerTest.java   |   1 +
 .../apache/beam/sdk/util/ReduceFnTester.java|   1 +
 .../beam/sdk/util/SimpleDoFnRunnerTest.java |   6 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   4 +-
 .../ImmutabilityCheckingBundleFactory.java  |   4 +-
 .../beam/runners/direct/ParDoEvaluator.java |   4 +-
 .../direct/ParDoMultiEvaluatorFactory.java  |  11 +-
 .../direct/ParDoSingleEvaluatorFactory.java |  11 +-
 .../direct/TransformEvaluatorFactory.java   |   6 +-
 .../direct/WriteWithShardingFactory.java|   4 +-
 .../ConsumerTrackingPipelineVisitorTest.java|  22 +-
 .../beam/runners/direct/DirectRunnerTest.java   |  24 +-
 .../ImmutabilityCheckingBundleFactoryTest.java  |   6 +-
 .../ImmutabilityEnforcementFactoryTest.java |   6 +-
 .../direct/KeyedPValueTrackingVisitorTest.java  |   6 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  |  10 +-
 .../direct/ParDoSingleEvaluatorFactoryTest.java |  10 +-
 .../runners/direct/WatermarkManagerTest.java|   7 +-
 .../beam/runners/flink/examples/TFIDF.java  |  16 +-
 .../beam/runners/flink/examples/WordCount.java  |   4 +-
 .../flink/examples/streaming/AutoComplete.java  |  16 +-
 .../flink/examples/streaming/JoinExamples.java  |   8 +-
 .../examples/streaming/KafkaIOExamples.java |   4 +-
 .../KafkaWindowedWordCountExample.java  |   6 +-
 .../examples/streaming/WindowedWordCount.java   |   6 +-
 .../FlinkBatchTransformTranslators.java |  12 +-
 .../FlinkStreamingTransformTranslators.java |   9 +-
 .../functions/FlinkDoFnFunction.java|  10 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   8 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  10 +-
 .../FlinkMultiOutputProcessContext.java |   6 +-
 .../functions/FlinkNoElementAssignContext.java  |   8 +-
 .../functions/FlinkPartialReduceFunction.java   |   8 +-
 .../functions/FlinkProcessContext.java  |  16 +-
 

[GitHub] incubator-beam pull request #758: [BEAM-498] Rename DoFn to OldDoFn, DoFnWit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/758


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[06/19] incubator-beam git commit: Rename DoFn to OldDoFn

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
index 3b314b2..8b00c03 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnWithContextTest.java
@@ -142,9 +142,9 @@ public class DoFnWithContextTest implements Serializable {
   @Test
   public void testDoFnWithContextUsingAggregators() {
 NoOpDoFn noOpFn = new NoOpDoFn<>();
-DoFn.Context context = noOpFn.context();
+OldDoFn.Context context = noOpFn.context();
 
-DoFn fn = spy(noOpFn);
+OldDoFn fn = spy(noOpFn);
 context = spy(context);
 
 @SuppressWarnings("unchecked")
@@ -225,7 +225,7 @@ public class DoFnWithContextTest implements Serializable {
   }
 
   /**
-   * Initialize a test pipeline with the specified {@link DoFn}.
+   * Initialize a test pipeline with the specified {@link OldDoFn}.
*/
   private  TestPipeline 
createTestPipeline(DoFnWithContext fn) {
 TestPipeline pipeline = TestPipeline.create();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
index 80825cb..b81eedb 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
@@ -130,7 +130,7 @@ public class FlattenTest implements Serializable {
 
 PCollection output = p
 .apply(Create.of((Void) null).withCoder(VoidCoder.of()))
-.apply(ParDo.withSideInputs(view).of(new DoFn() {
+.apply(ParDo.withSideInputs(view).of(new OldDoFn() {
   @Override
   public void processElement(ProcessContext c) {
 for (String side : c.sideInput(view)) {
@@ -339,7 +339,7 @@ public class FlattenTest implements Serializable {
 
   /
 
-  private static class IdentityFn extends DoFn {
+  private static class IdentityFn extends OldDoFn {
 @Override
 public void processElement(ProcessContext c) {
   c.output(c.element());

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64baf48/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index d6e4589..15c3ba8 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.transforms;
 
 import static org.apache.beam.sdk.TestUtils.KvMatcher.isKv;
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+
 import static org.hamcrest.CoreMatchers.equalTo;
 import static org.hamcrest.CoreMatchers.hasItem;
 import static org.hamcrest.Matchers.empty;
@@ -55,7 +56,6 @@ import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
-
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.junit.Assert;
@@ -371,7 +371,7 @@ public class GroupByKeyTest {
 pipeline.run();
   }
 
-  private static class AssertTimestamp extends DoFn, Void> {
+  private static class AssertTimestamp extends OldDoFn, Void> {
 private final Instant timestamp;
 
 public AssertTimestamp(Instant timestamp) {
@@ -506,7 +506,7 @@ public class GroupByKeyTest {
* Creates a KV that wraps the original KV together with a random key.
*/
   static class AssignRandomKey
-  extends DoFn, KV>> {
+  extends OldDoFn, KV>> {
 
 @Override
 public void processElement(ProcessContext c) throws Exception {

[jira] [Commented] (BEAM-522) Update FileSink.finalize_write() to be idempotent

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406952#comment-15406952
 ] 

Chamikara Jayalath commented on BEAM-522:
-

Actually, the bug is in the exists() implementation of gcsio.py.
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/gcsio.py#L237

Instead of catching IOError, we should be catching HttpError and checking error 
code to see if it's 404.

With  this fixed FileSink.finalize_write() becomes properly idempotent since we 
handle failures of rename() invocation at following location.
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L533

> Update FileSink.finalize_write() to be idempotent
> -
>
> Key: BEAM-522
> URL: https://issues.apache.org/jira/browse/BEAM-522
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently FileSink.finelize_write() in fileio.py [1] performs following 
> operations.
> (1) Obtains a list of temporary files as a side input
> (2) Renames each temporary file to the location where final output should be 
> stored.
> iobase.Sink.finalize_write() operation should be idempotent since runner 
> implementations may call this operation multiple times due to task failures. 
> Current implementation is not idempotent because if we re-run the operation 
> after renaming a sub-set of files, the operations may fail due to not being 
> able to find some files at source location (for example, [2] for GCS files).
> We can fix this by checking if the destination file is already available 
> before performing the rename and not performing the rename for files that are 
> already available at the destination.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L503
> [2] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/gcsio.py#L187
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-527) Pickling error when pickling a nested function

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-527:


 Summary: Pickling error when pickling a nested function 
 Key: BEAM-527
 URL: https://issues.apache.org/jira/browse/BEAM-527
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


There is a pickling error under the following conditions all happen:
- a function is defined inside a transforms' apply method
- then using it as MapFn
- that function references an instance variable of the outer transform.

Rewriting the nested function as an unnested DoFn appears to solve the problem.

If the limitations of pickling make it difficult to support nested functions 
then perhaps there's a way to make it easier for users to detect problems 
caused by nested functions and recommend appropriate fixes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-396) Coder.NonDeterministicException doesn't inherit from Exception

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-396.
--
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Coder.NonDeterministicException doesn't inherit from Exception
> --
>
> Key: BEAM-396
> URL: https://issues.apache.org/jira/browse/BEAM-396
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Chandni Singh
>Priority: Minor
>  Labels: findbugs, newbie, starter
> Fix For: 0.3.0-incubating
>
>
> [FindBugs 
> NM_CLASS_NOT_EXCEPTION|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L67]:
>  Class is not derived from an Exception, even though it is named as such.
> Applies to 
> [Coder.NonDeterministicException|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java#L263]
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-526) Mismatched pipelines give unclear error

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-526:


 Summary: Mismatched pipelines give unclear error
 Key: BEAM-526
 URL: https://issues.apache.org/jira/browse/BEAM-526
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


Mistakenly mixing two pipeline gives an unclear error. This is an error, 
however we should improve the error message.

This could be reproduced by trying to flatten two things from different 
pipelines.

Improve the message for this assert:
https://github.com/aaltay/incubator-beam/blob/python-sdk/sdks/python/apache_beam/transforms/util.py#L135




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-396) Coder.NonDeterministicException doesn't inherit from Exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406921#comment-15406921
 ] 

ASF GitHub Bot commented on BEAM-396:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/776


> Coder.NonDeterministicException doesn't inherit from Exception
> --
>
> Key: BEAM-396
> URL: https://issues.apache.org/jira/browse/BEAM-396
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Chandni Singh
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> NM_CLASS_NOT_EXCEPTION|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L67]:
>  Class is not derived from an Exception, even though it is named as such.
> Applies to 
> [Coder.NonDeterministicException|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java#L263]
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Make Coder.NonDeterministicException extend Exception instead of Throwable

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 23fa61ce0 -> 388816a80


Make Coder.NonDeterministicException extend Exception instead of Throwable


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fc35456c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fc35456c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fc35456c

Branch: refs/heads/master
Commit: fc35456cabee8556f2c400ba155ab603767654f5
Parents: da31a2d
Author: Chandni Singh 
Authored: Wed Aug 3 13:10:32 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 17:53:50 2016 -0700

--
 .../build-tools/src/main/resources/beam/findbugs-filter.xml | 5 -
 .../core/src/main/java/org/apache/beam/sdk/coders/Coder.java| 2 +-
 2 files changed, 1 insertion(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fc35456c/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index ee55ef0..28fd0df 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -30,11 +30,6 @@
   the issue.
 -->
   
-
-
-
-  
-  
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fc35456c/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
index 74af59e..136c00a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
@@ -260,7 +260,7 @@ public interface Coder extends Serializable {
* Exception thrown by {@link Coder#verifyDeterministic()} if the encoding is
* not deterministic, including details of why the encoding is not 
deterministic.
*/
-  public static class NonDeterministicException extends Throwable {
+  public static class NonDeterministicException extends Exception {
 private Coder coder;
 private List reasons;
 

[GitHub] incubator-beam pull request #776: BEAM-396 made Coder.NonDeterministic inher...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/776


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #776

[kenn]

This closes #776


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/388816a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/388816a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/388816a8

Branch: refs/heads/master
Commit: 388816a805446aa17ec73462c80b6d4f6546f419
Parents: 23fa61c fc35456
Author: Kenneth Knowles 
Authored: Wed Aug 3 17:54:53 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 17:54:53 2016 -0700

--
 .../build-tools/src/main/resources/beam/findbugs-filter.xml | 5 -
 .../core/src/main/java/org/apache/beam/sdk/coders/Coder.java| 2 +-
 2 files changed, 1 insertion(+), 6 deletions(-)
--

[4/4] incubator-beam git commit: This closes #754

[kenn]

This closes #754


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/23fa61ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/23fa61ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/23fa61ce

Branch: refs/heads/master
Commit: 23fa61ce0ce91371134b54618d2fc6ba0c84ef30
Parents: 636b5f7 8e7963f
Author: Kenneth Knowles 
Authored: Wed Aug 3 17:45:46 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 17:45:46 2016 -0700

--
 .../beam/runners/direct/CommittedResult.java|  23 ++-
 .../beam/runners/direct/CompletionCallback.java |   8 +
 .../beam/runners/direct/EvaluationContext.java  |  11 +-
 .../direct/ExecutorServiceParallelExecutor.java | 163 ---
 .../runners/direct/StepTransformResult.java |  20 ++-
 .../beam/runners/direct/TransformExecutor.java  |   1 +
 .../beam/runners/direct/TransformResult.java|  11 ++
 .../runners/direct/ViewEvaluatorFactory.java|   9 +-
 .../runners/direct/CommittedResultTest.java |  26 ++-
 .../runners/direct/EvaluationContextTest.java   |   3 +
 .../runners/direct/StepTransformResultTest.java |  91 +++
 .../runners/direct/TransformExecutorTest.java   |  33 +++-
 .../runners/direct/WatermarkManagerTest.java|  11 +-
 13 files changed, 336 insertions(+), 74 deletions(-)
--

[jira] [Commented] (BEAM-156) Implement Quiescence Signalling in the InProcessPipelineRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406911#comment-15406911
 ] 

ASF GitHub Bot commented on BEAM-156:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/754


> Implement Quiescence Signalling in the InProcessPipelineRunner
> --
>
> Key: BEAM-156
> URL: https://issues.apache.org/jira/browse/BEAM-156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> A pipeline is quiescent when the following two properties hold:
>   There are no triggers that can fire, given the current processing time and 
> watermark
>   All pending elements cannot make progress until a side input produces 
> additional output
> This is approximately equivalent to: If no more input is received, the 
> pipeline will not perform any additional processing absent advances in 
> processing time or event time
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] incubator-beam git commit: Add ProducedOutput method to TransformResult

[kenn]

Add ProducedOutput method to TransformResult

This can communicate that a PTransform that produced no outputs still
should cause pending work to be evaluated. PCollectionViews modifiy the
state of the evaluator and can cause formerly blocked PTransforms to be
able to progress.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f7cc7e17
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f7cc7e17
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f7cc7e17

Branch: refs/heads/master
Commit: f7cc7e178db211509aecb65ba203930fd159629a
Parents: a8eb274
Author: Thomas Groh 
Authored: Tue Jul 26 09:53:22 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 17:45:12 2016 -0700

--
 .../beam/runners/direct/CommittedResult.java| 16 +++-
 .../beam/runners/direct/EvaluationContext.java  |  3 +-
 .../runners/direct/StepTransformResult.java | 28 ++-
 .../beam/runners/direct/TransformResult.java|  9 +++
 .../runners/direct/ViewEvaluatorFactory.java|  4 +-
 .../runners/direct/CommittedResultTest.java | 24 --
 .../runners/direct/StepTransformResultTest.java | 85 
 .../runners/direct/TransformExecutorTest.java   |  5 +-
 .../runners/direct/WatermarkManagerTest.java|  7 +-
 9 files changed, 163 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7cc7e17/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
index e86f07d..e9a40a8 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.direct;
 
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 
 import com.google.auto.value.AutoValue;
 
@@ -49,12 +50,23 @@ abstract class CommittedResult {
*/
   public abstract Iterable> getOutputs();
 
+  /**
+   * Returns if the transform that produced this result produced outputs.
+   *
+   * Transforms that produce output via modifying the state of the runner 
(e.g.
+   * {@link CreatePCollectionView}) should explicitly set this to true. If 
{@link #getOutputs()}
+   * returns a nonempty iterable, this will also return true.
+   */
+  public abstract boolean producedOutputs();
+
   public static CommittedResult create(
   TransformResult original,
   CommittedBundle unprocessedElements,
-  Iterable> outputs) {
+  Iterable> outputs,
+  boolean producedOutputs) {
 return new AutoValue_CommittedResult(original.getTransform(),
 unprocessedElements,
-outputs);
+outputs,
+producedOutputs);
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7cc7e17/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index ea713fa..610a62d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -159,7 +159,8 @@ class EvaluationContext {
 completedBundle == null
 ? null
 : completedBundle.withElements((Iterable) 
result.getUnprocessedElements()),
-committedBundles);
+committedBundles,
+result.producedOutput());
 watermarkManager.updateWatermarks(
 completedBundle,
 result.getTimerUpdate().withCompletedTimers(completedTimers),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7cc7e17/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
index 176bb14..3d6841d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
+++ 

[3/4] incubator-beam git commit: Use the State of the Executor to drive progress

[kenn]

Use the State of the Executor to drive progress

Add the concept of Quiescence to ExecutorServiceParallelExecutor.

If the executor is Quiescent, it should interrogate root nodes for
additional work. If not, runs of the monitor should update the state as
appropriate.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8e7963f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8e7963f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8e7963f4

Branch: refs/heads/master
Commit: 8e7963f498955f10db1867e106ee82068b5f0c2a
Parents: f7cc7e1
Author: Thomas Groh 
Authored: Fri Jul 22 13:47:43 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 17:45:12 2016 -0700

--
 .../beam/runners/direct/CommittedResult.java|  11 +-
 .../beam/runners/direct/CompletionCallback.java |   5 +-
 .../beam/runners/direct/EvaluationContext.java  |  10 +-
 .../direct/ExecutorServiceParallelExecutor.java | 108 ---
 .../runners/direct/StepTransformResult.java |  26 ++---
 .../beam/runners/direct/TransformExecutor.java  |   2 +-
 .../beam/runners/direct/TransformResult.java|  12 ++-
 .../runners/direct/ViewEvaluatorFactory.java|   9 +-
 .../runners/direct/CommittedResultTest.java |  10 +-
 .../runners/direct/EvaluationContextTest.java   |   3 +
 .../runners/direct/StepTransformResultTest.java |  26 +++--
 .../runners/direct/TransformExecutorTest.java   |  11 +-
 .../runners/direct/WatermarkManagerTest.java|   6 +-
 13 files changed, 176 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8e7963f4/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
index e9a40a8..7e0cd8e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
@@ -24,6 +24,8 @@ import 
org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 
 import com.google.auto.value.AutoValue;
 
+import java.util.Set;
+
 import javax.annotation.Nullable;
 
 /**
@@ -57,16 +59,21 @@ abstract class CommittedResult {
* {@link CreatePCollectionView}) should explicitly set this to true. If 
{@link #getOutputs()}
* returns a nonempty iterable, this will also return true.
*/
-  public abstract boolean producedOutputs();
+  public abstract Set getProducedOutputTypes();
 
   public static CommittedResult create(
   TransformResult original,
   CommittedBundle unprocessedElements,
   Iterable> outputs,
-  boolean producedOutputs) {
+  Set producedOutputs) {
 return new AutoValue_CommittedResult(original.getTransform(),
 unprocessedElements,
 outputs,
 producedOutputs);
   }
+
+  enum OutputType {
+PCOLLECTION_VIEW,
+BUNDLE
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8e7963f4/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
index 2f496e9..8e51d6f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
@@ -18,6 +18,7 @@
 package org.apache.beam.runners.direct;
 
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
+import org.apache.beam.sdk.transforms.AppliedPTransform;
 
 /**
  * A callback for completing a bundle of input.
@@ -31,8 +32,10 @@ interface CompletionCallback {
 
   /**
* Handle an input bundle that did not require processing.
+   *
+   * This occurs when a Source has no splits that can currently produce 
outputs.
*/
-  void handleEmpty(CommittedBundle inputBundle);
+  void handleEmpty(AppliedPTransform transform);
 
   /**
* Handle a result that terminated abnormally due to the provided {@link 
Throwable}.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8e7963f4/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
--
diff --git 

[GitHub] incubator-beam pull request #754: [BEAM-156] Use Quiescence to Drive the Dir...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/754


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/4] incubator-beam git commit: Add handleEmpty to CompletionCallback

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 636b5f7f6 -> 23fa61ce0


Add handleEmpty to CompletionCallback

This is invoked when a Transform Executor has no work to do. Usually
this is due to reinvocation of a Source.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a8eb274a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a8eb274a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a8eb274a

Branch: refs/heads/master
Commit: a8eb274aca3e4ab2b89a3e3d44bb2c755fd638eb
Parents: 8b1e64a
Author: Thomas Groh 
Authored: Fri Jul 22 18:01:41 2016 -0700
Committer: Thomas Groh 
Committed: Fri Jul 29 16:00:47 2016 -0700

--
 .../beam/runners/direct/CompletionCallback.java |  5 ++
 .../direct/ExecutorServiceParallelExecutor.java | 57 +---
 .../beam/runners/direct/TransformExecutor.java  |  1 +
 .../runners/direct/TransformExecutorTest.java   | 27 ++
 4 files changed, 47 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8eb274a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
index 0c5fe24..2f496e9 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
@@ -30,6 +30,11 @@ interface CompletionCallback {
   CommittedBundle inputBundle, TransformResult result);
 
   /**
+   * Handle an input bundle that did not require processing.
+   */
+  void handleEmpty(CommittedBundle inputBundle);
+
+  /**
* Handle a result that terminated abnormally due to the provided {@link 
Throwable}.
*/
   void handleThrowable(CommittedBundle inputBundle, Throwable t);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8eb274a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 43195e3..3901472 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -122,7 +122,8 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
 this.visibleUpdates = new ArrayBlockingQueue<>(20);
 
 parallelExecutorService = 
TransformExecutorServices.parallel(executorService);
-defaultCompletionCallback = new DefaultCompletionCallback();
+defaultCompletionCallback =
+new 
TimerIterableCompletionCallback(Collections.emptyList());
   }
 
   private CacheLoader
@@ -217,18 +218,19 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
   /**
* The base implementation of {@link CompletionCallback} that provides 
implementations for
* {@link #handleResult(CommittedBundle, TransformResult)} and
-   * {@link #handleThrowable(CommittedBundle, Throwable)}, given an 
implementation of
-   * {@link #getCommittedResult(CommittedBundle, TransformResult)}.
+   * {@link #handleThrowable(CommittedBundle, Throwable)}.
*/
-  private abstract class CompletionCallbackBase implements CompletionCallback {
-protected abstract CommittedResult getCommittedResult(
-CommittedBundle inputBundle,
-TransformResult result);
+  private class TimerIterableCompletionCallback implements CompletionCallback {
+private final Iterable timers;
+
+protected TimerIterableCompletionCallback(Iterable timers) {
+  this.timers = timers;
+}
 
 @Override
 public final CommittedResult handleResult(
 CommittedBundle inputBundle, TransformResult result) {
-  CommittedResult committedResult = getCommittedResult(inputBundle, 
result);
+  CommittedResult committedResult = 
evaluationContext.handleResult(inputBundle, timers, result);
   for (CommittedBundle outputBundle : committedResult.getOutputs()) {
 allUpdates.offer(ExecutorUpdate.fromBundle(outputBundle,
 valueToConsumers.get(outputBundle.getPCollection(;
@@ -242,43 +244,12 @@ final class 

[jira] [Created] (BEAM-525) Verify that ParDo with multiple outputs with tags un declared in with_outputs() work

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-525:


 Summary: Verify that ParDo with multiple outputs with tags un 
declared in with_outputs() work 
 Key: BEAM-525
 URL: https://issues.apache.org/jira/browse/BEAM-525
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


test_undeclared_side_outputs was failing (when last checked) under certain 
conditions:

See this TODO:
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/dataflow_test.py#L202

This is probably not failing any more but it needs to be verified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-524) Description of "type" argument in Aggregator is incorrect

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-524:
-
Component/s: sdk-py

> Description of "type" argument in Aggregator is incorrect
> -
>
> Key: BEAM-524
> URL: https://issues.apache.org/jira/browse/BEAM-524
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Priority: Minor
>
> Two problems with documentation for "type" argument.
> Trivial:  Remove "by default".  This phrase implies that there are other 
> alternatives besides what is listed.  There aren't.
> Non trivial.  The documentation says "types appropriate to the combine_fn" 
> are accepted.  I tried 
> Accumulator("foo", max, datetime.datetime)
> This failed even though "datetime.datetime" is a perfectly reasonable type to 
> want to take the max of.  (I wanted to know precisely when the last job 
> finished.)
> Either the documentation needs to be changed to specify that max/min only 
> apply to numeric types, or the code needs to be changed to allow other uses 
> of min and max.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-523) Minor typo in aggregator_test.py

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-523:
-
   Assignee: (was: Frances Perry)
 Labels: starter  (was: )
Component/s: (was: beam-model)
 sdk-py

> Minor typo in aggregator_test.py
> 
>
> Key: BEAM-523
> URL: https://issues.apache.org/jira/browse/BEAM-523
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Priority: Trivial
>  Labels: starter
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> aggregators is repeatedly misspelled as aggeregators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #774: Run integration tests in parallel

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/774


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Run integration tests in parallel

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ca15cac0a -> 636b5f7f6


Run integration tests in parallel


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/790f82b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/790f82b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/790f82b0

Branch: refs/heads/master
Commit: 790f82b008b9e3ad9fccaf5ce985ee18a1e248c4
Parents: ca15cac
Author: Dan Halperin 
Authored: Tue Aug 2 23:51:45 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 17:13:56 2016 -0700

--
 examples/java/pom.xml  | 2 ++
 sdks/java/io/google-cloud-platform/pom.xml | 2 ++
 2 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/790f82b0/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 8b98288..d0a6b34 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -187,6 +187,8 @@
   verify
 
 
+  all
+  4
   
 
${integrationTestPipelineOptions}
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/790f82b0/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index dcd1d5c..1596a66 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -78,6 +78,8 @@
   verify
 
 
+  all
+  4
   
 
${integrationTestPipelineOptions}
   

[2/2] incubator-beam git commit: Closes #774

[dhalperi]

Closes #774


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/636b5f7f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/636b5f7f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/636b5f7f

Branch: refs/heads/master
Commit: 636b5f7f6bbcf53d7af85ba2a43f30cce31e0379
Parents: ca15cac 790f82b
Author: Dan Halperin 
Authored: Wed Aug 3 17:13:57 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 17:13:57 2016 -0700

--
 examples/java/pom.xml  | 2 ++
 sdks/java/io/google-cloud-platform/pom.xml | 2 ++
 2 files changed, 4 insertions(+)
--

[GitHub] incubator-beam pull request #777: Bind checkstyle:check to the test-compile ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/777


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #777

[dhalperi]

Closes #777


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ca15cac0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ca15cac0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ca15cac0

Branch: refs/heads/master
Commit: ca15cac0a12760f7eaaa84166e2e921484a85466
Parents: da31a2d 1fecb28
Author: Dan Halperin 
Authored: Wed Aug 3 17:04:30 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 17:04:30 2016 -0700

--
 pom.xml | 8 
 1 file changed, 8 insertions(+)
--

[1/2] incubator-beam git commit: Bind checkstyle:check to the test-compile phase to fail earlier

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master da31a2d77 -> ca15cac0a


Bind checkstyle:check to the test-compile phase to fail earlier


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1fecb284
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1fecb284
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1fecb284

Branch: refs/heads/master
Commit: 1fecb2844c0b7e6cdedb93d93164d2151c847f98
Parents: da31a2d
Author: Kenneth Knowles 
Authored: Wed Aug 3 16:12:15 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 3 16:12:15 2016 -0700

--
 pom.xml | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1fecb284/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3b98382..23b7e4d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -703,7 +703,15 @@
 true
   
   
+
 
+  test-compile
   
 check
   

[jira] [Created] (BEAM-524) Description of "type" argument in Aggregator is incorrect

[Frank Yellin (JIRA)]

Frank Yellin created BEAM-524:
-

 Summary: Description of "type" argument in Aggregator is incorrect
 Key: BEAM-524
 URL: https://issues.apache.org/jira/browse/BEAM-524
 Project: Beam
  Issue Type: Bug
Reporter: Frank Yellin
Priority: Minor


Two problems with documentation for "type" argument.

Trivial:  Remove "by default".  This phrase implies that there are other 
alternatives besides what is listed.  There aren't.

Non trivial.  The documentation says "types appropriate to the combine_fn" are 
accepted.  I tried 
Accumulator("foo", max, datetime.datetime)
This failed even though "datetime.datetime" is a perfectly reasonable type to 
want to take the max of.  (I wanted to know precisely when the last job 
finished.)

Either the documentation needs to be changed to specify that max/min only apply 
to numeric types, or the code needs to be changed to allow other uses of min 
and max.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-523) Minor typo in aggregator_test.py

[Frank Yellin (JIRA)]

Frank Yellin created BEAM-523:
-

 Summary: Minor typo in aggregator_test.py
 Key: BEAM-523
 URL: https://issues.apache.org/jira/browse/BEAM-523
 Project: Beam
  Issue Type: Bug
  Components: beam-model
Reporter: Frank Yellin
Assignee: Frances Perry
Priority: Trivial


aggregators is repeatedly misspelled as aggeregators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #778: Correct some accidental renames

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/778

Correct some accidental renames

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

IDE over-eagerly replaced some occurrences of createAggregator with
createAggregatorForDoFn. This corrects that.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam fix-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/778.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #778


commit a39d682e3e2c22eb1efa597dc734d44c27113937
Author: bchambers 
Date:   2016-08-03T20:38:43Z

Correct some accidental renames

IDE over-eagerly replaced some occurrences of createAggregator with
createAggregatorForDoFn. This corrects that.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #777: Bind checkstyle:check to the test-compile ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/777

Bind checkstyle:check to the test-compile phase to fail earlier

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Today we run checkstyle _after packaging_ so we have to wait for all the 
various jars to be built before we fail.

This change binds checkstyle to the earliest phase that seems reasonable.

R: anyone

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam checkstyle-earlier

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/777.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #777


commit 1fecb2844c0b7e6cdedb93d93164d2151c847f98
Author: Kenneth Knowles 
Date:   2016-08-03T23:12:15Z

Bind checkstyle:check to the test-compile phase to fail earlier




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-396) Coder.NonDeterministicException doesn't inherit from Exception

[Chandni Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chandni Singh reassigned BEAM-396:
--

Assignee: Chandni Singh

> Coder.NonDeterministicException doesn't inherit from Exception
> --
>
> Key: BEAM-396
> URL: https://issues.apache.org/jira/browse/BEAM-396
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Chandni Singh
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> NM_CLASS_NOT_EXCEPTION|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L67]:
>  Class is not derived from an Exception, even though it is named as such.
> Applies to 
> [Coder.NonDeterministicException|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java#L263]
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-396) Coder.NonDeterministicException doesn't inherit from Exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406643#comment-15406643
 ] 

ASF GitHub Bot commented on BEAM-396:
-

GitHub user chandnisingh opened a pull request:

https://github.com/apache/incubator-beam/pull/776

BEAM-396 made Coder.NonDeterministic inherit from Exception and not T…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…hrowable

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chandnisingh/incubator-beam BEAM-396

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/776.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #776






> Coder.NonDeterministicException doesn't inherit from Exception
> --
>
> Key: BEAM-396
> URL: https://issues.apache.org/jira/browse/BEAM-396
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> NM_CLASS_NOT_EXCEPTION|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L67]:
>  Class is not derived from an Exception, even though it is named as such.
> Applies to 
> [Coder.NonDeterministicException|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java#L263]
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #776: BEAM-396 made Coder.NonDeterministic inher...

[chandnisingh]

GitHub user chandnisingh opened a pull request:

https://github.com/apache/incubator-beam/pull/776

BEAM-396 made Coder.NonDeterministic inherit from Exception and not T…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…hrowable

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chandnisingh/incubator-beam BEAM-396

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/776.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #776






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-522) Update FileSink.finalize_write() to be idempotent

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-522:
---

 Summary: Update FileSink.finalize_write() to be idempotent
 Key: BEAM-522
 URL: https://issues.apache.org/jira/browse/BEAM-522
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Currently FileSink.finelize_write() in fileio.py [1] performs following 
operations.

(1) Obtains a list of temporary files as a side input
(2) Renames each temporary file to the location where final output should be 
stored.

iobase.Sink.finalize_write() operation should be idempotent since runner 
implementations may call this operation multiple times due to task failures. 

Current implementation is not idempotent because if we re-run the operation 
after renaming a sub-set of files, the operations may fail due to not being 
able to find some files at source location (for example, [2] for GCS files).

We can fix this by checking if the destination file is already available before 
performing the rename and not performing the rename for files that are already 
available at the destination.

[1] 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L503

[2] 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/gcsio.py#L187
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: Implement add_input for all CombineFns.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e834fa82b -> 65152cab8


Implement add_input for all CombineFns.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3ebf28c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3ebf28c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3ebf28c6

Branch: refs/heads/python-sdk
Commit: 3ebf28c6e0d17af3720076e33f88a0f126a89059
Parents: e834fa8
Author: Robert Bradshaw 
Authored: Tue Jul 26 01:15:55 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 2 15:52:28 2016 -0700

--
 sdks/python/apache_beam/transforms/combiners.py | 16 
 sdks/python/apache_beam/transforms/core.py  |  6 +++---
 2 files changed, 11 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3ebf28c6/sdks/python/apache_beam/transforms/combiners.py
--
diff --git a/sdks/python/apache_beam/transforms/combiners.py 
b/sdks/python/apache_beam/transforms/combiners.py
index 155dcc6..c3f0da1 100644
--- a/sdks/python/apache_beam/transforms/combiners.py
+++ b/sdks/python/apache_beam/transforms/combiners.py
@@ -132,6 +132,9 @@ class CountCombineFn(core.CombineFn):
   def create_accumulator(self):
 return 0
 
+  def add_input(self, accumulator, element):
+return accumulator + 1
+
   def add_inputs(self, accumulator, elements):
 return accumulator + len(elements)
 
@@ -425,9 +428,9 @@ class _TupleCombineFnBase(core.CombineFn):
 
 class TupleCombineFn(_TupleCombineFnBase):
 
-  def add_inputs(self, accumulator, elements):
-return [c.add_inputs(a, e)
-for c, a, e in zip(self._combiners, accumulator, zip(*elements))]
+  def add_input(self, accumulator, element):
+return [c.add_input(a, e)
+for c, a, e in zip(self._combiners, accumulator, element)]
 
   def with_common_input(self):
 return SingleInputTupleCombineFn(*self._combiners)
@@ -435,8 +438,8 @@ class TupleCombineFn(_TupleCombineFnBase):
 
 class SingleInputTupleCombineFn(_TupleCombineFnBase):
 
-  def add_inputs(self, accumulator, elements):
-return [c.add_inputs(a, elements)
+  def add_input(self, accumulator, element):
+return [c.add_input(a, element)
 for c, a in zip(self._combiners, accumulator)]
 
 
@@ -522,9 +525,6 @@ def curry_combine_fn(fn, args, kwargs):
   def add_input(self, accumulator, element):
 return fn.add_input(accumulator, element, *args, **kwargs)
 
-  def add_inputs(self, accumulator, elements):
-return fn.add_inputs(accumulator, elements, *args, **kwargs)
-
   def merge_accumulators(self, accumulators):
 return fn.merge_accumulators(accumulators, *args, **kwargs)
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3ebf28c6/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 38b9cd2..da26205 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -270,7 +270,7 @@ class CombineFn(WithTypeHints):
   1. Input values are partitioned into one or more batches.
   2. For each batch, the create_accumulator method is invoked to create a fresh
  initial "accumulator" value representing the combination of zero values.
-  3. For each input value in the batch, the add_inputs method is invoked to
+  3. For each input value in the batch, the add_input method is invoked to
  combine more values with the accumulator for that batch.
   4. The merge_accumulators method is invoked to combine accumulators from
  separate batches into a single combined output accumulator value, once all
@@ -296,7 +296,7 @@ class CombineFn(WithTypeHints):
   def add_input(self, accumulator, element, *args, **kwargs):
 """Return result of folding element into accumulator.
 
-CombineFn implementors must override either add_input or add_inputs.
+CombineFn implementors must override add_input.
 
 Args:
   accumulator: the current accumulator
@@ -420,7 +420,7 @@ class CallableWrapperCombineFn(CombineFn):
 if accumulator is self._EMPTY:
   return self._fn(elements, *args, **kwargs)
 elif isinstance(elements, (list, tuple)):
-  return self._fn([accumulator] + elements, *args, **kwargs)
+  return self._fn([accumulator] + list(elements), *args, **kwargs)
 else:
   def union():
 yield accumulator

[2/3] incubator-beam git commit: Document TupleCombineFns

[robertwb]

Document TupleCombineFns


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4a2239d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4a2239d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4a2239d3

Branch: refs/heads/python-sdk
Commit: 4a2239d3701e13622998c71107d263c8966e73e1
Parents: 3ebf28c
Author: Robert Bradshaw 
Authored: Wed Aug 3 13:52:36 2016 -0700
Committer: Robert Bradshaw 
Committed: Wed Aug 3 13:52:36 2016 -0700

--
 sdks/python/apache_beam/transforms/combiners.py | 12 
 1 file changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4a2239d3/sdks/python/apache_beam/transforms/combiners.py
--
diff --git a/sdks/python/apache_beam/transforms/combiners.py 
b/sdks/python/apache_beam/transforms/combiners.py
index c3f0da1..a0604b8 100644
--- a/sdks/python/apache_beam/transforms/combiners.py
+++ b/sdks/python/apache_beam/transforms/combiners.py
@@ -427,6 +427,12 @@ class _TupleCombineFnBase(core.CombineFn):
 
 
 class TupleCombineFn(_TupleCombineFnBase):
+  """A combiner for combining tuples via a tuple of combiners.
+
+  Takes as input a tuple of N CombineFns and combines N-tuples by
+  combining the k-th element of each tuple with the k-th CombineFn,
+  outputting a new N-tuple of combined values.
+  """
 
   def add_input(self, accumulator, element):
 return [c.add_input(a, e)
@@ -437,6 +443,12 @@ class TupleCombineFn(_TupleCombineFnBase):
 
 
 class SingleInputTupleCombineFn(_TupleCombineFnBase):
+  """A combiner for combining a single value via a tuple of combiners.
+
+  Takes as input a tuple of N CombineFns and combines elements by
+  applying each CombineFn to each input, producing an N-tuple of
+  the outputs corresponding to each of the N CombineFn's outputs.
+  """
 
   def add_input(self, accumulator, element):
 return [c.add_input(a, element)

[3/3] incubator-beam-site git commit: Renegerate website after changes

[dhalperi]

Renegerate website after changes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/342fe7e4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/342fe7e4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/342fe7e4

Branch: refs/heads/asf-site
Commit: 342fe7e42910ae5d0d999a6e185ab7a026fc01c3
Parents: 045e004
Author: Dan Halperin 
Authored: Wed Aug 3 12:02:26 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 12:02:43 2016 -0700

--
 content/blog/2016/08/03/six-months.html | 215 +++
 content/blog/index.html |  16 ++
 content/feed.xml|  55 -
 content/index.html  |   2 +
 .../learn/runners/capability-matrix/index.html  |   2 +-
 5 files changed, 287 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/342fe7e4/content/blog/2016/08/03/six-months.html
--
diff --git a/content/blog/2016/08/03/six-months.html 
b/content/blog/2016/08/03/six-months.html
new file mode 100644
index 000..2a3c48f
--- /dev/null
+++ b/content/blog/2016/08/03/six-months.html
@@ -0,0 +1,215 @@
+
+
+
+  
+  
+  
+  
+
+  Apache Beam: Six Months in Incubation
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/blog/2016/08/03/six-months.html; 
data-proofer-ignore>
+  http://beam.incubator.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+ Use 
+ 
+ User Hub
+ 
+ General
+ Beam 
Overview
+ Quickstart  
+ Release 
Information
+ 
+ Example Walkthroughs
+ WordCount
+ Mobile 
Gaming
+ 
+ Support
+ Mailing 
Lists
+  Issue Tracking
+ http://stackoverflow.com/questions/tagged/apache-beam;>Beam on 
StackOverflow
+  http://apachebeam.slack.com;>Beam Slack 
Channel
+ 
+   
+
+ Learn 
+ 
+ Learner Hub
+ 
+ Beam Concepts
+ Programming 
Guide
+ Presentation Materials
+ Additional 
Resources
+ 
+ SDKs
+ Java SDK
+ Java SDK API 
Reference
+ 
+ Runners
+ Capability Matrix
+ Direct 
Runner
+ Apache Flink 
Runner
+ Apache Spark 
Runner
+ Cloud Dataflow 
Runner
+ 
+   
+
+ Contribute 
+ 
+ Contributor Hub
+ 
+ Basics
+ Contribution Guide
+ Mailing 
Lists
+  Source 
Repository
+  Issue Tracking
+  
+ Technical Resources
+ https://goo.gl/nk5OM0;>Technical 
Vision
+ 
+   
+Blog
+
+  Project
+  
+Logos and design
+Public Meetings
+   Team
+  
+
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
+  
+http://www.apache.org/;>ASF Homepage
+http://www.apache.org/licenses/;>License
+http://www.apache.org/security/;>Security
+http://www.apache.org/foundation/thanks.html;>Thanks
+

[2/3] incubator-beam-site git commit: Update to today

[dhalperi]

Update to today


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/045e0040
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/045e0040
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/045e0040

Branch: refs/heads/asf-site
Commit: 045e0040d1c82eac90661b45d37f136cf6f6a08c
Parents: 4244fe2
Author: Dan Halperin 
Authored: Wed Aug 3 12:00:43 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 12:00:43 2016 -0700

--
 _posts/2016-08-03-six-months.md | 43 
 _posts/2016-08-04-six-months.md | 43 
 2 files changed, 43 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/045e0040/_posts/2016-08-03-six-months.md
--
diff --git a/_posts/2016-08-03-six-months.md b/_posts/2016-08-03-six-months.md
new file mode 100644
index 000..1141eaf
--- /dev/null
+++ b/_posts/2016-08-03-six-months.md
@@ -0,0 +1,43 @@
+---
+layout: post
+title:  "Apache Beam: Six Months in Incubation"
+date:   2016-08-03 00:00:01 -0700
+excerpt_separator: 
+categories: blog
+authors:
+  - fjp
+---
+
+It’s been just over six months since Apache Beam was formally accepted into 
incubation with the [Apache Software Foundation](http://www.apache.org). As a 
community, we’ve been hard at work getting Beam off the ground.
+
+
+
+Looking just at raw numbers for those first six months, that’s:
+
+* 48,238 lines of preexisting code donated by Cloudera, dataArtisans, and 
Google.
+* 761 pull requests from 45 contributors.
+* 498 Jira issues opened and 245 resolved.
+* 1 incubating release (and another 1 in progress). 
+* 4,200 hours of automated tests. 
+* 161 subscribers / 606 messages on user@.
+* 217 subscribers / 1205 messages on dev@.
+* 277 stars and 174 forks on GitHub.
+
+And behind those numbers, there’s been a ton of technical progress, 
including:
+
+* Refactoring of the entire codebase, examples, and tests to be truly 
runner-independent.
+* New functionality in the Apache Flink runner for timestamps/windows in batch 
and bounded sources and side inputs in streaming mode.
+* Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
+* Several new runners from the wider Apache community -- Apache Gearpump has 
its own feature branch, Apache Apex has a PR, and conversations are starting on 
Apache Storm and others.
+* New SDKs/DSLs for exposing the Beam model -- the Python SDK from Google is 
in on a feature branch, and there are plans to add the Scio DSL from Spotify.
+* Support for additional data sources and sinks -- Apache Kafka and JMS are 
in, there are PRs for Amazon Kinesis, Apache Cassandra, and MongoDB, and more 
connectors are being planned.
+
+But perhaps most importantly, we’re committed to building an involved, 
welcoming community. So far, we’ve:
+
+* Started building a vibrant developer community, with detailed design 
discussions on features like DoFn reuse semantics, serialization technology, 
and an API for accessing state.
+* Started building a user community with an active mailing list and 
improvements to the website and documentation.
+* Had multiple talks on Beam at venues including ApacheCon, Hadoop Summit, 
Kafka Summit, JBCN Barcelona, and Strata.
+* Presented at multiple existing meetups and are starting to organize some of 
our own.
+
+While it’s nice to reflect back on all we’ve done, we’re working full 
_stream_ ahead towards a stable release and graduation from incubator. And 
we’d love your help -- join the [mailing 
lists](http://beam.incubator.apache.org/use/mailing-lists/), check out the 
[contribution 
guide](http://beam.incubator.apache.org/contribute/contribution-guide/), and 
grab a [starter 
task](https://issues.apache.org/jira/browse/BEAM-520?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20in%20(newbie%2C%20starter))
 from Jira!
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/045e0040/_posts/2016-08-04-six-months.md
--
diff --git a/_posts/2016-08-04-six-months.md b/_posts/2016-08-04-six-months.md
deleted file mode 100644
index c4ed246..000
--- a/_posts/2016-08-04-six-months.md
+++ /dev/null
@@ -1,43 +0,0 @@

-layout: post
-title:  "Apache Beam: Six Months in Incubation"
-date:   2016-08-04 00:00:01 -0700
-excerpt_separator: 
-categories: blog
-authors:
-  - fjp

-
-It’s been just over six months since Apache Beam was formally accepted into 
incubation with the [Apache Software Foundation](http://www.apache.org). As a 
community, we’ve been 

[1/3] incubator-beam-site git commit: Added a half birthday blog post, as discussed on dev@.

[dhalperi]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site c630ee0ed -> 342fe7e42


Added a half birthday blog post, as discussed on dev@.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4244fe2b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4244fe2b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4244fe2b

Branch: refs/heads/asf-site
Commit: 4244fe2b9638b973d99ab73bad9eace7f79cb070
Parents: c630ee0
Author: Frances Perry 
Authored: Tue Aug 2 22:36:07 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 3 11:59:04 2016 -0700

--
 _posts/2016-08-04-six-months.md | 43 
 1 file changed, 43 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4244fe2b/_posts/2016-08-04-six-months.md
--
diff --git a/_posts/2016-08-04-six-months.md b/_posts/2016-08-04-six-months.md
new file mode 100644
index 000..c4ed246
--- /dev/null
+++ b/_posts/2016-08-04-six-months.md
@@ -0,0 +1,43 @@
+---
+layout: post
+title:  "Apache Beam: Six Months in Incubation"
+date:   2016-08-04 00:00:01 -0700
+excerpt_separator: 
+categories: blog
+authors:
+  - fjp
+---
+
+It’s been just over six months since Apache Beam was formally accepted into 
incubation with the [Apache Software Foundation](http://www.apache.org). As a 
community, we’ve been hard at work getting Beam off the ground.
+
+
+
+Looking just at raw numbers for those first six months, that’s:
+
+* 48,238 lines of preexisting code donated by Cloudera, dataArtisans, and 
Google.
+* 761 pull requests from 45 contributors.
+* 498 Jira issues opened and 245 resolved.
+* 1 incubating release (and another 1 in progress). 
+* 4,200 hours of automated tests. 
+* 161 subscribers / 606 messages on user@.
+* 217 subscribers / 1205 messages on dev@.
+* 277 stars and 174 forks on GitHub.
+
+And behind those numbers, there’s been a ton of technical progress, 
including:
+
+* Refactoring of the entire codebase, examples, and tests to be truly 
runner-independent.
+* New functionality in the Apache Flink runner for timestamps/windows in batch 
and bounded sources and side inputs in streaming mode.
+* Work in progress to upgrade the Apache Spark runner to use Spark 2.0.
+* Several new runners from the wider Apache community -- Apache Gearpump has 
its own feature branch, Apache Apex has a PR, and conversations are starting on 
Apache Storm and others.
+* New SDKs/DSLs for exposing the Beam model -- the Python SDK from Google is 
in on a feature branch, and there are plans to add the Scio DSL from Spotify.
+* Support for additional data sources and sinks -- Apache Kafka and JMS are 
in, there are PRs for Amazon Kinesis, Apache Cassandra, and MongoDB, and more 
connectors are being planned.
+
+But perhaps most importantly, we’re committed to building an involved, 
welcoming community. So far, we’ve:
+
+* Started building a vibrant developer community, with detailed design 
discussions on features like DoFn reuse semantics, serialization technology, 
and an API for accessing state.
+* Started building a user community with an active mailing list and 
improvements to the website and documentation.
+* Had multiple talks on Beam at venues including ApacheCon, Hadoop Summit, 
Kafka Summit, JBCN Barcelona, and Strata.
+* Presented at multiple existing meetups and are starting to organize some of 
our own.
+
+While it’s nice to reflect back on all we’ve done, we’re working full 
_stream_ ahead towards a stable release and graduation from incubator. And 
we’d love your help -- join the [mailing 
lists](http://beam.incubator.apache.org/use/mailing-lists/), check out the 
[contribution 
guide](http://beam.incubator.apache.org/contribute/contribution-guide/), and 
grab a [starter 
task](https://issues.apache.org/jira/browse/BEAM-520?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20in%20(newbie%2C%20starter))
 from Jira!
+

[jira] [Commented] (BEAM-516) Update navigation for Javadoc

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406558#comment-15406558
 ] 

Ismaël Mejía commented on BEAM-516:
---

Just for reference I include a link to a previous (and related) issue related 
to the javadocs:
https://issues.apache.org/jira/browse/BEAM-385

I think that having the javadocs as part of the CI systems (like flink does on 
the mentioned JIRA) brings some advantages, e.g. up to date docs for master, as 
well as an independent from the website management (to not to make some of the 
changes by hand).


> Update navigation for Javadoc 
> --
>
> Key: BEAM-516
> URL: https://issues.apache.org/jira/browse/BEAM-516
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Ismaël Mejía
>Assignee: James Malone
>Priority: Minor
> Attachments: screenshot.png
>
>
> The link to the latest version of the java documentation dissapeared with the 
> recent changes to the website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-499) Remove unused code in apiclient.py and iobase.py

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-499.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Removed unused code from apiclient.py.

Seems like iobase.py currently cannot be further trimmed down.

> Remove unused code in apiclient.py and iobase.py
> 
>
> Key: BEAM-499
> URL: https://issues.apache.org/jira/browse/BEAM-499
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> There is some code in apiclient.py and iobase.py that is not used by Dataflow 
> SDK. This code has to be removed.
> E.g.:
> class DataflowWorkerClient
> def reader_progress_to_cloud_progress() and other similar methods.
> def splits_to_split_response()
> class ConcatPosition



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-502) Properly handle None/null in json conversions

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-502.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Properly handle None/null in json conversions
> -
>
> Key: BEAM-502
> URL: https://issues.apache.org/jira/browse/BEAM-502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> json_value.py  has to be updated to properly handle JSON to/from Python 
> 'None' conversions.
> For example, currently writing a dictionary of the form {'aa': 'value', 
> 'bb':None} using BigQuery sink fails when using DirectPipelineRunner since we 
> do not properly handle 'None' values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-521) Execute some file-based reads via composite transform instead of custom source

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-521:
-

 Summary: Execute some file-based reads via composite transform 
instead of custom source
 Key: BEAM-521
 URL: https://issues.apache.org/jira/browse/BEAM-521
 Project: Beam
  Issue Type: Improvement
Reporter: Eugene Kirpichov


The BoundedSource API is intended for cases where the source can provide 
meaningfull progress, dynamic splitting and size estimation. E.g. it's a good 
fit for processing a moderate number of large files, or a key-value table.

However, existing runners have scalability limitations on how many bundles a 
BoundedSource can split into, and this leads to it being a very poor fit for 
the case of processing many small files: the source ends up splitting in a too 
large number of bundles (at least 1 per file) overwhelming the runner.

This is a frequent use case, and the power of BoundedSource API is not needed 
in this case: small files don't need to be dynamically split, progress 
estimation is not needed, and size estimation is a "nice-to-have" but not 
entirely necessary.

In this case, it'd be better to execute the read not as a raw 
Read.from(BoundedSource) executed natively by the runner, but as a 
ParDo(splitIntoBundles) + fusion break + ParDo(read each bundle). That way the 
bundles end up as a simple PCollection with no scalability limitations, and 
most likely much smaller per-bundle overhead.

Implementation options:
- The BoundedSource API could provide a hint method telling Read.from() to 
expand in this way
- Individual connectors, such as TextIO.Read, could switch between expanding 
into Read.from() or into this composite transform depending on parameters (e.g. 
TextIO.Read.withCompressionType(GZ) would always expand into the composite 
transform, because for compressed files BoundedSource API is unnecessary)
- Something else?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-521) Execute some bounded source reads via composite transform

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-521:
--
Summary: Execute some bounded source reads via composite transform  (was: 
Execute some file-based reads via composite transform instead of custom source)

> Execute some bounded source reads via composite transform
> -
>
> Key: BEAM-521
> URL: https://issues.apache.org/jira/browse/BEAM-521
> Project: Beam
>  Issue Type: Improvement
>Reporter: Eugene Kirpichov
>
> The BoundedSource API is intended for cases where the source can provide 
> meaningfull progress, dynamic splitting and size estimation. E.g. it's a good 
> fit for processing a moderate number of large files, or a key-value table.
> However, existing runners have scalability limitations on how many bundles a 
> BoundedSource can split into, and this leads to it being a very poor fit for 
> the case of processing many small files: the source ends up splitting in a 
> too large number of bundles (at least 1 per file) overwhelming the runner.
> This is a frequent use case, and the power of BoundedSource API is not needed 
> in this case: small files don't need to be dynamically split, progress 
> estimation is not needed, and size estimation is a "nice-to-have" but not 
> entirely necessary.
> In this case, it'd be better to execute the read not as a raw 
> Read.from(BoundedSource) executed natively by the runner, but as a 
> ParDo(splitIntoBundles) + fusion break + ParDo(read each bundle). That way 
> the bundles end up as a simple PCollection with no scalability limitations, 
> and most likely much smaller per-bundle overhead.
> Implementation options:
> - The BoundedSource API could provide a hint method telling Read.from() to 
> expand in this way
> - Individual connectors, such as TextIO.Read, could switch between expanding 
> into Read.from() or into this composite transform depending on parameters 
> (e.g. TextIO.Read.withCompressionType(GZ) would always expand into the 
> composite transform, because for compressed files BoundedSource API is 
> unnecessary)
> - Something else?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #775: New

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam/pull/775

New

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam new

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/775.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #775


commit c4a2441e933dd49a5dee5a7295e29f43cd403de2
Author: Ismaël Mejía 
Date:   2016-08-03T09:03:43Z

Remove useless semicolons

commit ea8e4e7415dcbe0b8939de788d2713b28c5a511e
Author: Ismaël Mejía 
Date:   2016-08-03T09:22:40Z

Remove unneeded java keywords/validations and fix Filter style

commit 2e21b8b9b013a1f112ffb73833432b9b356ffa8b
Author: Ismaël Mejía 
Date:   2016-08-03T09:37:37Z

Fix invalid Javadoc references and some other documentation issues

commit 72ffd759e3c29d83e1a880bc499482ccfb603f70
Author: Ismaël Mejía 
Date:   2016-08-03T10:10:39Z

Add rules for unused semicolons and overcomplicated boolean expresions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #774: Run integration tests in parallel

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/774

Run integration tests in parallel

WIP

CC: @kennknowles 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
integration-test-parallel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/774.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #774


commit 3d2ecbaaa90bb6910096423ac87e882f5d649e9e
Author: Dan Halperin 
Date:   2016-08-03T06:51:45Z

Run integration tests in parallel




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #773: Miscellaneous fixups

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/773


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #773

[dhalperi]

Closes #773


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/da31a2d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/da31a2d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/da31a2d7

Branch: refs/heads/master
Commit: da31a2d77467e1ecbd23963b67ab8e2eb7ffec38
Parents: 63c5d19 68c5be4
Author: Dan Halperin 
Authored: Tue Aug 2 23:35:19 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:35:19 2016 -0700

--
 pom.xml  | 2 +-
 .../beam/runners/core/UnboundedReadFromBoundedSourceTest.java| 2 +-
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java   | 1 -
 .../org/apache/beam/runners/dataflow/DataflowPipelineJob.java| 2 +-
 .../src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java  | 1 -
 .../src/main/java/org/apache/beam/sdk/options/GcpOptions.java| 1 -
 .../src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java | 4 ++--
 .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java| 2 +-
 8 files changed, 6 insertions(+), 9 deletions(-)
--

[1/2] incubator-beam git commit: Miscellaneous fixups

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 63c5d19b3 -> da31a2d77


Miscellaneous fixups

* Remove a tab in pom.xml
* Fix some javadoc
* Remove some newlines
* Remove some commented-out code
* Remove an accidental System.err.println left in
* Remove unnecessary boxing/unboxing


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/68c5be43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/68c5be43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/68c5be43

Branch: refs/heads/master
Commit: 68c5be43828b7b34629c0799a40b56c11f1add4f
Parents: 63c5d19
Author: Dan Halperin 
Authored: Tue Aug 2 18:37:50 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:35:12 2016 -0700

--
 pom.xml  | 2 +-
 .../beam/runners/core/UnboundedReadFromBoundedSourceTest.java| 2 +-
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java   | 1 -
 .../org/apache/beam/runners/dataflow/DataflowPipelineJob.java| 2 +-
 .../src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java  | 1 -
 .../src/main/java/org/apache/beam/sdk/options/GcpOptions.java| 1 -
 .../src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java | 4 ++--
 .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java| 2 +-
 8 files changed, 6 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68c5be43/pom.xml
--
diff --git a/pom.xml b/pom.xml
index caec6d5..3b98382 100644
--- a/pom.xml
+++ b/pom.xml
@@ -573,7 +573,7 @@
   
 io.netty
 netty-handler
-   ${netty.version}
+${netty.version}
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68c5be43/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
index dfbc675..45c1414 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
@@ -360,7 +360,7 @@ public class UnboundedReadFromBoundedSourceTest {
 if (channel.read(buff) != 1) {
   return false;
 }
-current = new Byte(buff.get(0));
+current = buff.get(0);
 offset += 1;
 return true;
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68c5be43/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index 5d3ab3f..477da30 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -109,7 +109,6 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
   DoFn, KV> gabwDoFn =
   GroupAlsoByWindowViaWindowSetDoFn.create(
   windowingStrategy,
-  // new DirectStateInternalsFactory(stepContext),
   new ConstantStateInternalsFactory(stateInternals),
   SystemReduceFn.buffering(valueCoder));
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68c5be43/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index a02d280..3194f7c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -113,7 +113,7 @@ public class DataflowPipelineJob implements PipelineResult {
*
 

[jira] [Closed] (BEAM-337) Cleanup NOTICE file

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-337.

   Resolution: Fixed
Fix Version/s: (was: 0.2.0-incubating)
   Not applicable

> Cleanup NOTICE file
> ---
>
> Key: BEAM-337
> URL: https://issues.apache.org/jira/browse/BEAM-337
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Affects Versions: 0.1.0-incubating
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> NOTICE file contain unneeded text (i.e mentions  Apache v2.0 licence). There 
> no need to generally mention Apache 2.0 licences in NOTICE according to:
> http://www.apache.org/dev/licensing-howto.html#alv2-dep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-514) Add all mandatory links

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-514.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Add all mandatory links
> ---
>
> Key: BEAM-514
> URL: https://issues.apache.org/jira/browse/BEAM-514
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Frances Perry
> Fix For: Not applicable
>
>
> Except from: 
> http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E
> > Branding wise I think you are missing a few of the
> required links [3] including a link back to the Apache homepage.
> http://www.apache.org/foundation/marks/pmcs.html#navigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-514) Add all mandatory links

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-514.


> Add all mandatory links
> ---
>
> Key: BEAM-514
> URL: https://issues.apache.org/jira/browse/BEAM-514
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Frances Perry
> Fix For: Not applicable
>
>
> Except from: 
> http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E
> > Branding wise I think you are missing a few of the
> required links [3] including a link back to the Apache homepage.
> http://www.apache.org/foundation/marks/pmcs.html#navigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-515) Add feature logo and incubator logo

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-515.

   Resolution: Fixed
Fix Version/s: Not applicable

> Add feature logo and incubator logo
> ---
>
> Key: BEAM-515
> URL: https://issues.apache.org/jira/browse/BEAM-515
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Frances Perry
>Priority: Critical
> Fix For: Not applicable
>
>
> Except from: 
> http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E
>  A feather ASF logo would be a nice addition as well. [4]
> http://www.apache.org/foundation/press/kit/#links
> While we're in there, I believe we still need to add the Apache Incubator egg 
> logo. http://incubator.apache.org/images/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-369) Add travis config to website

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-369.

   Resolution: Fixed
Fix Version/s: Not applicable

> Add travis config to website
> 
>
> Key: BEAM-369
> URL: https://issues.apache.org/jira/browse/BEAM-369
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: James Malone
>Assignee: James Malone
> Fix For: Not applicable
>
>
> Add a Travis CI config to the Beam website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam-site git commit: Renegerate website after changes

[dhalperi]

Renegerate website after changes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c630ee0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c630ee0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c630ee0e

Branch: refs/heads/asf-site
Commit: c630ee0ed5d2353f6b7eccc6d2bb27cc584041b1
Parents: 0799aab
Author: Dan Halperin 
Authored: Tue Aug 2 23:24:31 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:24:31 2016 -0700

--
 content/beam/capability/2016/03/17/capability-matrix.html  | 2 +-
 content/beam/capability/2016/04/03/presentation-materials.html | 2 +-
 content/beam/python/sdk/2016/02/25/python-sdk-now-public.html  | 2 +-
 content/beam/release/2016/06/15/first-release.html | 2 +-
 content/beam/update/website/2016/02/22/beam-has-a-logo.html| 2 +-
 content/blog/2016/05/18/splitAtFraction-method.html| 2 +-
 content/blog/2016/05/27/where-is-my-pcollection-dot-map.html   | 2 +-
 content/blog/2016/06/13/flink-batch-runner-milestone.html  | 2 +-
 content/blog/index.html| 2 +-
 content/coming-soon.html   | 2 +-
 content/contribute/contribution-guide/index.html   | 2 +-
 content/contribute/index.html  | 2 +-
 content/contribute/source-repository/index.html| 2 +-
 content/feed.xml   | 4 ++--
 content/index.html | 2 +-
 content/learn/index.html   | 2 +-
 content/learn/presentation-materials/index.html| 2 +-
 content/learn/programming-guide/index.html | 2 +-
 content/learn/resources/index.html | 2 +-
 content/learn/runners/capability-matrix/index.html | 4 ++--
 content/learn/runners/dataflow/index.html  | 2 +-
 content/learn/runners/direct/index.html| 2 +-
 content/learn/runners/flink/index.html | 2 +-
 content/learn/runners/index.html   | 2 +-
 content/learn/runners/spark/index.html | 2 +-
 content/learn/sdks/index.html  | 2 +-
 content/learn/sdks/java/index.html | 2 +-
 content/learn/sdks/javadoc/index.html  | 2 +-
 content/privacy_policy/index.html  | 2 +-
 content/project/logos/index.html   | 2 +-
 content/project/public-meetings/index.html | 2 +-
 content/project/team/index.html| 2 +-
 content/use/beam-overview/index.html   | 2 +-
 content/use/index.html | 2 +-
 content/use/issue-tracking/index.html  | 2 +-
 content/use/mailing-lists/index.html   | 2 +-
 content/use/quickstart/index.html  | 2 +-
 content/use/releases/index.html| 2 +-
 content/use/walkthroughs/index.html| 2 +-
 39 files changed, 41 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c630ee0e/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index 89457d3..480598f 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -114,7 +114,7 @@
   
 
   
-  
+  
 
   https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c630ee0e/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index 7e1c84a..66b26f5 100644
--- a/content/beam/capability/2016/04/03/presentation-materials.html
+++ b/content/beam/capability/2016/04/03/presentation-materials.html
@@ -114,7 +114,7 @@
   
 
   
-  
+  
 
   https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" 

[1/2] incubator-beam-site git commit: Fix ASF header collapsing

[dhalperi]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 749776a87 -> c630ee0ed


Fix ASF header collapsing


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0799aab4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0799aab4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0799aab4

Branch: refs/heads/asf-site
Commit: 0799aab46f041435390b913ce6fc8ddccb106482
Parents: 749776a
Author: Dan Halperin 
Authored: Tue Aug 2 23:24:19 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:24:19 2016 -0700

--
 _includes/header.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0799aab4/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 1f1c933..d7e31db 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -81,7 +81,7 @@
   
 
   
-  
+  
 
   https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
   

[1/2] incubator-beam-site git commit: Minor style tweaks for alignment

[dhalperi]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 706bfba71 -> 749776a87


Minor style tweaks for alignment


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/18a0ace5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/18a0ace5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/18a0ace5

Branch: refs/heads/asf-site
Commit: 18a0ace5824f3c8264ec6206f3d6d1e0436a
Parents: 706bfba
Author: Dan Halperin 
Authored: Tue Aug 2 23:21:36 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:21:36 2016 -0700

--
 _includes/header.html | 2 +-
 _layouts/default.html | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/18a0ace5/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 8d80f96..1f1c933 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -84,7 +84,7 @@
   
 
   https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
-  
+  
 http://www.apache.org/;>ASF Homepage
 http://www.apache.org/licenses/;>License
 http://www.apache.org/security/;>Security

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/18a0ace5/_layouts/default.html
--
diff --git a/_layouts/default.html b/_layouts/default.html
index 4d7663e..b7b9c90 100644
--- a/_layouts/default.html
+++ b/_layouts/default.html
@@ -9,7 +9,7 @@
 
 
 
-  
+  
 {{ content }}
   
 

[2/2] incubator-beam-site git commit: Renegerate website after changes

[dhalperi]

Renegerate website after changes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/749776a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/749776a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/749776a8

Branch: refs/heads/asf-site
Commit: 749776a87b87a71b56d3a831593a7ec51c43867c
Parents: 18a0ace
Author: Dan Halperin 
Authored: Tue Aug 2 23:21:57 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:21:57 2016 -0700

--
 content/beam/capability/2016/03/17/capability-matrix.html  | 4 ++--
 content/beam/capability/2016/04/03/presentation-materials.html | 4 ++--
 content/beam/python/sdk/2016/02/25/python-sdk-now-public.html  | 4 ++--
 content/beam/release/2016/06/15/first-release.html | 4 ++--
 content/beam/update/website/2016/02/22/beam-has-a-logo.html| 4 ++--
 content/blog/2016/05/18/splitAtFraction-method.html| 4 ++--
 content/blog/2016/05/27/where-is-my-pcollection-dot-map.html   | 4 ++--
 content/blog/2016/06/13/flink-batch-runner-milestone.html  | 4 ++--
 content/blog/index.html| 4 ++--
 content/coming-soon.html   | 4 ++--
 content/contribute/contribution-guide/index.html   | 4 ++--
 content/contribute/index.html  | 4 ++--
 content/contribute/source-repository/index.html| 4 ++--
 content/feed.xml   | 4 ++--
 content/index.html | 4 ++--
 content/learn/index.html   | 4 ++--
 content/learn/presentation-materials/index.html| 4 ++--
 content/learn/programming-guide/index.html | 4 ++--
 content/learn/resources/index.html | 4 ++--
 content/learn/runners/capability-matrix/index.html | 6 +++---
 content/learn/runners/dataflow/index.html  | 4 ++--
 content/learn/runners/direct/index.html| 4 ++--
 content/learn/runners/flink/index.html | 4 ++--
 content/learn/runners/index.html   | 4 ++--
 content/learn/runners/spark/index.html | 4 ++--
 content/learn/sdks/index.html  | 4 ++--
 content/learn/sdks/java/index.html | 4 ++--
 content/learn/sdks/javadoc/index.html  | 4 ++--
 content/privacy_policy/index.html  | 4 ++--
 content/project/logos/index.html   | 4 ++--
 content/project/public-meetings/index.html | 4 ++--
 content/project/team/index.html| 4 ++--
 content/use/beam-overview/index.html   | 4 ++--
 content/use/index.html | 4 ++--
 content/use/issue-tracking/index.html  | 4 ++--
 content/use/mailing-lists/index.html   | 4 ++--
 content/use/quickstart/index.html  | 4 ++--
 content/use/releases/index.html| 4 ++--
 content/use/walkthroughs/index.html| 4 ++--
 39 files changed, 79 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/749776a8/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index 44763b2..89457d3 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -117,7 +117,7 @@
   
 
   https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
-  
+  
 http://www.apache.org/;>ASF Homepage
 http://www.apache.org/licenses/;>License
 http://www.apache.org/security/;>Security
@@ -136,7 +136,7 @@
 
 
 
-  
+  
 
 
 http://schema.org/BlogPosting;>

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/749776a8/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index 2ccded3..7e1c84a 100644
--- 

[jira] [Commented] (BEAM-514) Add all mandatory links

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405394#comment-15405394
 ] 

ASF GitHub Bot commented on BEAM-514:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/33


> Add all mandatory links
> ---
>
> Key: BEAM-514
> URL: https://issues.apache.org/jira/browse/BEAM-514
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Frances Perry
>
> Except from: 
> http://mail-archives.apache.org/mod_mbox/incubator-general/201608.mbox/%3C7E0226B1-0386-499C-8473-61A8E51A691B%40classsoftware.com%3E
> > Branding wise I think you are missing a few of the
> required links [3] including a link back to the Apache homepage.
> http://www.apache.org/foundation/marks/pmcs.html#navigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #33: [BEAM-514][BEAM-515] Fixed Apache logo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/33


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/6] incubator-beam-site git commit: Closes #33

[dhalperi]

Closes #33


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d29d0761
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d29d0761
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d29d0761

Branch: refs/heads/asf-site
Commit: d29d076191577059425e759dcc6243cd5949dd30
Parents: cc13a69 9b57c29
Author: Dan Halperin 
Authored: Tue Aug 2 23:05:19 2016 -0700
Committer: Dan Halperin 
Committed: Tue Aug 2 23:05:19 2016 -0700

--
 _includes/header.html | 11 ++-
 index.md  | 13 ++---
 2 files changed, 16 insertions(+), 8 deletions(-)
--

[1/6] incubator-beam-site git commit: Addressed BEAM-514 (add required asf links)

[dhalperi]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site cc13a697e -> 706bfba71


Addressed BEAM-514 (add required asf links)


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/2c975326
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/2c975326
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/2c975326

Branch: refs/heads/asf-site
Commit: 2c97532679a5a2f32ea6495f7cafbad946aa5321
Parents: cc13a69
Author: Frances Perry 
Authored: Tue Aug 2 21:33:35 2016 -0700
Committer: Frances Perry 
Committed: Tue Aug 2 21:33:35 2016 -0700

--
 _includes/header.html | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2c975326/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index d78c77f..83d2c44 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -78,7 +78,16 @@
 Logos and 
design
 Public 
Meetings
Team
-http://apache.org/licenses/LICENSE-2.0.html;>License
+  
+
+
+  Apache Software 
Foundation
+  
+http://www.apache.org/;>ASF Homepage
+http://www.apache.org/licenses/;>License
+http://www.apache.org/security/;>Security
+http://www.apache.org/foundation/thanks.html;>Thanks
+http://www.apache.org/foundation/sponsorship.html;>Sponsorship