date:20160728

[incubator-beam] Git Push Summary

2016-07-28 Thread dhalperi

Repository: incubator-beam
Updated Tags:  refs/tags/v0.2.0-incubating-RC2 [created] bf6d9af75

incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.2.0-incubating-RC2

2016-07-28 Thread dhalperi

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.2.0-incubating 7ec5e16c6 -> 6901ff648


[maven-release-plugin] rollback changes from release preparation of 
v0.2.0-incubating-RC2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6901ff64
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6901ff64
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6901ff64

Branch: refs/heads/release-0.2.0-incubating
Commit: 6901ff648de3fdcf11fffe2fbb429aef13eacfb8
Parents: 7ec5e16
Author: Dan Halperin 
Authored: Thu Jul 28 01:03:00 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 01:03:00 2016 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/examples/pom.xml  | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/flink/runner/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hdfs/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/microbenchmarks/pom.xml   | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 28 files changed, 29 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index cdc1296..5044393 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.2.0-incubating
+0.2.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index ef081b9..36d44c5 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.2.0-incubating
+0.2.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index beaa34a..ee12b69 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.2.0-incubating
+0.2.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/pom.xml
--
diff --git a/pom.xml b/pom.xml
index be28223..8e8ef47 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.2.0-incubating
+  0.2.0-incubating-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-v0.2.0-incubating-RC2
+release-0.2.0-incubating
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index bc42860..fc8be0a 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-runners-parent
-0.2.0-incubating
+0.2.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6901ff64/runners/direct-java/pom.xml

[1/2] incubator-beam git commit: jms: fix artifact ID in pom.xml

2016-07-28 Thread dhalperi

Repository: incubator-beam
Updated Branches:
  refs/heads/master af396bf8e -> ff873f4eb


jms: fix artifact ID in pom.xml


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b4e651b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b4e651b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b4e651b4

Branch: refs/heads/master
Commit: b4e651b4686e498c1d0d529a878e9f6c911356e0
Parents: af396bf
Author: Dan Halperin 
Authored: Wed Jul 27 23:18:35 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jul 27 23:18:35 2016 -0700

--
 sdks/java/io/jms/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b4e651b4/sdks/java/io/jms/pom.xml
--
diff --git a/sdks/java/io/jms/pom.xml b/sdks/java/io/jms/pom.xml
index 9cdc7f4..7113434 100644
--- a/sdks/java/io/jms/pom.xml
+++ b/sdks/java/io/jms/pom.xml
@@ -26,7 +26,7 @@
 ../pom.xml
   
 
-  jms
+  beam-sdks-java-io-jms
   Apache Beam :: SDKs :: Java :: IO :: JMS
   IO to read and write to JMS (Java Messaging Service)
 destinations (queues and topics). 
@@ -131,4 +131,4 @@
 
   
 
-
\ No newline at end of file
+

[GitHub] incubator-beam pull request #750: [BEAM-79] Merge branch 'master' into gearp...

2016-07-28 Thread manuzhang

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/750

[BEAM-79] Merge branch 'master' into gearpump_runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #750


commit 34467f92d5a31b47f95b734c737fde0a8277311b
Author: Thomas Groh 
Date:   2016-07-15T17:51:24Z

Make TransformEvaluatorFactory reuse Explicit

Transform Evaluator Factories must be reused for the entire execution of
a Pipeline and must not be reused across pipelines.

Remove EvaluatorKey, and key explicitly by the transform application.

commit ad6ee01825740ee47f25ca036fa5f352375bbe6e
Author: Dan Halperin 
Date:   2016-07-19T06:40:13Z

Checkstyle: enforce package-info.java presence

Ignores tests and examples

commit c4ad11832235eef0b73d299cd02d1a224c130ece
Author: Dan Halperin 
Date:   2016-07-20T16:16:28Z

Closes #692

commit 6d7efe3df8cde590722d94441eca3922a3a67734
Author: Kenneth Knowles 
Date:   2016-07-20T17:30:55Z

This closes #666

commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d
Author: Thomas Groh 
Date:   2016-06-30T17:06:52Z

Add withAllowedLateness with Closing Behavior to Window

This makes the static constructors for withAllowedLateness symmetric to
the PTransform builder methods. It also allows references to
Window#withAllowedLateness(Duration, ClosingBehavior).

commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e
Author: Kenneth Knowles 
Date:   2016-07-20T17:40:13Z

This closes #567

commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4
Author: Thomas Groh 
Date:   2016-06-28T22:44:49Z

Use the ParDo Application to Cache DoFns

A DoFn application is the scope of reuse.

Factor CloningThreadLocal as the top-level class instead of
SerializableCloningThreadLocalCacheLoader, and extract the Fn from the
AppliedPTransform when loading an absent element.

commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254
Author: Kenneth Knowles 
Date:   2016-07-20T17:55:53Z

This closes #554

commit b240525affb205a83054577233f3a4a508fe1c72
Author: Dan Halperin 
Date:   2016-07-18T19:05:02Z

BigQueryIO: move to google-cloud-platform module

* Move package from io to io.gcp.bigquery
* Move from SDK core into GCP-IO module
* Fixup references and import orders
* Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils
* Rewrite a unit test in sdk core to not depend on BigQueryIO
* Fixup Javadoc in SDK core that need not depend on BigQueryIO
* Make utility classes package-private

commit 7ec8781a2e18548a23c882329f0b50f7254202ec
Author: Dan Halperin 
Date:   2016-07-20T20:02:43Z

Closes #681

commit bdb65278873a5010a625dc6a569ba25b17374c06
Author: Kenneth Knowles 
Date:   2016-07-20T04:36:10Z

Add os-maven-plugin to Spark runner

commit 84332ee9716233af928e85c14c534714ab828531
Author: Chandni Singh 
Date:   2016-07-20T00:30:16Z

BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9
Author: Luke Cwik 
Date:   2016-07-21T13:34:52Z

[BEAM-372] added a test that verifies if a coder consumes bytes equal to 
encoded bytes

This closes #695

commit ae2144196c351cc2ee544e030d793929d8607696
Author: Dan Halperin 
Date:   2016-07-20T17:26:02Z

BigtableIO: upgrade to 0.9.1

* Use the uber jar
* Remove OS classifier mumbo jumbo
* Move common dependency versioning to root pom

commit 0eb81fdc15bc3dce28e24e73427bd389e0f6d60d
Author: Luke Cwik 
Date:   2016-07-21T15:26:41Z

[BEAM-445] BigtableIO: upgrade to 0.9.1

This closes #701

commit

[jira] [Commented] (BEAM-79) Gearpump runner

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397251#comment-15397251
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/750

[BEAM-79] Merge branch 'master' into gearpump_runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #750


commit 34467f92d5a31b47f95b734c737fde0a8277311b
Author: Thomas Groh 
Date:   2016-07-15T17:51:24Z

Make TransformEvaluatorFactory reuse Explicit

Transform Evaluator Factories must be reused for the entire execution of
a Pipeline and must not be reused across pipelines.

Remove EvaluatorKey, and key explicitly by the transform application.

commit ad6ee01825740ee47f25ca036fa5f352375bbe6e
Author: Dan Halperin 
Date:   2016-07-19T06:40:13Z

Checkstyle: enforce package-info.java presence

Ignores tests and examples

commit c4ad11832235eef0b73d299cd02d1a224c130ece
Author: Dan Halperin 
Date:   2016-07-20T16:16:28Z

Closes #692

commit 6d7efe3df8cde590722d94441eca3922a3a67734
Author: Kenneth Knowles 
Date:   2016-07-20T17:30:55Z

This closes #666

commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d
Author: Thomas Groh 
Date:   2016-06-30T17:06:52Z

Add withAllowedLateness with Closing Behavior to Window

This makes the static constructors for withAllowedLateness symmetric to
the PTransform builder methods. It also allows references to
Window#withAllowedLateness(Duration, ClosingBehavior).

commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e
Author: Kenneth Knowles 
Date:   2016-07-20T17:40:13Z

This closes #567

commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4
Author: Thomas Groh 
Date:   2016-06-28T22:44:49Z

Use the ParDo Application to Cache DoFns

A DoFn application is the scope of reuse.

Factor CloningThreadLocal as the top-level class instead of
SerializableCloningThreadLocalCacheLoader, and extract the Fn from the
AppliedPTransform when loading an absent element.

commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254
Author: Kenneth Knowles 
Date:   2016-07-20T17:55:53Z

This closes #554

commit b240525affb205a83054577233f3a4a508fe1c72
Author: Dan Halperin 
Date:   2016-07-18T19:05:02Z

BigQueryIO: move to google-cloud-platform module

* Move package from io to io.gcp.bigquery
* Move from SDK core into GCP-IO module
* Fixup references and import orders
* Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils
* Rewrite a unit test in sdk core to not depend on BigQueryIO
* Fixup Javadoc in SDK core that need not depend on BigQueryIO
* Make utility classes package-private

commit 7ec8781a2e18548a23c882329f0b50f7254202ec
Author: Dan Halperin 
Date:   2016-07-20T20:02:43Z

Closes #681

commit bdb65278873a5010a625dc6a569ba25b17374c06
Author: Kenneth Knowles 
Date:   2016-07-20T04:36:10Z

Add os-maven-plugin to Spark runner

commit 84332ee9716233af928e85c14c534714ab828531
Author: Chandni Singh 
Date:   2016-07-20T00:30:16Z

BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9
Author: Luke Cwik 
Date:   2016-07-21T13:34:52Z

[BEAM-372] added a test that verifies if a coder consumes bytes equal to 
encoded bytes

This closes #695

commit ae2144196c351cc2ee544e030d793929d8607696
Author: Dan Halperin 
Date:   2016-07-20T17:26:02Z

BigtableIO: upgrade to 0.9.1

* Use the uber jar
* Remove OS classifier mumbo jumbo
* Move common dependency versioning to

[2/2] incubator-beam git commit: Closes #749

2016-07-28 Thread dhalperi

Closes #749


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ff873f4e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ff873f4e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ff873f4e

Branch: refs/heads/master
Commit: ff873f4eb1c90a871db0c0e32b0cf964c88a154e
Parents: af396bf b4e651b
Author: Dan Halperin 
Authored: Thu Jul 28 00:03:51 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 00:03:51 2016 -0700

--
 sdks/java/io/jms/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #749: jms: fix artifact ID in pom.xml

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/749


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: jms: fix artifact ID in pom.xml

2016-07-28 Thread dhalperi

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.2.0-incubating a6633756e -> 7ec5e16c6


jms: fix artifact ID in pom.xml


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/448dcaea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/448dcaea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/448dcaea

Branch: refs/heads/release-0.2.0-incubating
Commit: 448dcaeab9ca5b46e10e7d6d2846cb994cea76ac
Parents: a663375
Author: Dan Halperin 
Authored: Wed Jul 27 23:18:35 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 00:04:30 2016 -0700

--
 sdks/java/io/jms/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/448dcaea/sdks/java/io/jms/pom.xml
--
diff --git a/sdks/java/io/jms/pom.xml b/sdks/java/io/jms/pom.xml
index fa56c7b..ffb9da2 100644
--- a/sdks/java/io/jms/pom.xml
+++ b/sdks/java/io/jms/pom.xml
@@ -26,7 +26,7 @@
 ../pom.xml
   
 
-  jms
+  beam-sdks-java-io-jms
   Apache Beam :: SDKs :: Java :: IO :: JMS
   IO to read and write to JMS (Java Messaging Service)
 destinations (queues and topics). 
@@ -131,4 +131,4 @@
 
   
 
-
\ No newline at end of file
+

[2/2] incubator-beam git commit: [maven-release-plugin] prepare release v0.2.0-incubating-RC2

2016-07-28 Thread dhalperi

[maven-release-plugin] prepare release v0.2.0-incubating-RC2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7ec5e16c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7ec5e16c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7ec5e16c

Branch: refs/heads/release-0.2.0-incubating
Commit: 7ec5e16c654044c395c1ce0930f3e3ac13102701
Parents: 448dcae
Author: Dan Halperin 
Authored: Thu Jul 28 01:02:44 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 01:02:44 2016 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/examples/pom.xml  | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/flink/runner/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hdfs/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/microbenchmarks/pom.xml   | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 28 files changed, 29 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 5044393..cdc1296 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.2.0-incubating-SNAPSHOT
+0.2.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 36d44c5..ef081b9 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.2.0-incubating-SNAPSHOT
+0.2.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index ee12b69..beaa34a 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.2.0-incubating-SNAPSHOT
+0.2.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8e8ef47..be28223 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.2.0-incubating-SNAPSHOT
+  0.2.0-incubating
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-release-0.2.0-incubating
+v0.2.0-incubating-RC2
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index fc8be0a..bc42860 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-runners-parent
-0.2.0-incubating-SNAPSHOT
+0.2.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ec5e16c/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml

[jira] [Closed] (BEAM-491) Reuse context and disable UI in the Spark runner tests

2016-07-28 Thread Amit Sela (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-491.
--
Resolution: Fixed

Closed by https://github.com/apache/incubator-beam/pull/736

> Reuse context and disable UI in the Spark runner tests 
> ---
>
> Key: BEAM-491
> URL: https://issues.apache.org/jira/browse/BEAM-491
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.1.0-incubating
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Currently, only RunnableOnService tests reuse the context for some reason, 
> although surefire is using 1 fork (no reuse). I don't see a reason why not to 
> reuse in all surefire executions.
> UI could be disabled for test executions as well. This could also help with 
> Jenkins ports issues we've been experiencing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-349) Spark runner should provide a default BoM

2016-07-28 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-349:
--
Fix Version/s: (was: 0.2.0-incubating)

> Spark runner should provide a default BoM
> -
>
> Key: BEAM-349
> URL: https://issues.apache.org/jira/browse/BEAM-349
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> When an user defines a pipeline, he has to define the following dependencies 
> in order to work with the Spark runner (in addition of the Spark runner 
> artifact itself):
> {code}
>   
> org.apache.spark
> spark-core_2.10
> ${spark.version}
> 
> 
> org.apache.spark
> spark-streaming_2.10
> ${spark.version}
> 
> 
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
> 
> 
> com.fasterxml.jackson.core
> jackson-annotations
> ${jackson.version}
> 
> 
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
> 
> 
> com.fasterxml.jackson.module
> jackson-module-scala_2.10
> ${jackson.version}
> 
> {code}
> These dependencies should come as transitive dependencies from the Spark 
> runner (as Flink or dataflow runners do). It means we should change the scope 
> of the dependencies in the Spark runner pom to create a BoM (Bill of 
> Material).
> I will experiment a bit (with [~amitsela]).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Use AutoValue for StepTransformResult

2016-07-28 Thread lcwik

Use AutoValue for StepTransformResult

This closes #746


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a46081eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a46081eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a46081eb

Branch: refs/heads/master
Commit: a46081eb5e36c79f78711b8927e03753822a8fd2
Parents: ff873f4 19de0b8
Author: Luke Cwik 
Authored: Thu Jul 28 10:13:04 2016 -0400
Committer: Luke Cwik 
Committed: Thu Jul 28 10:13:04 2016 -0400

--
 .../runners/direct/StepTransformResult.java | 76 
 1 file changed, 15 insertions(+), 61 deletions(-)
--

[jira] [Commented] (BEAM-156) Implement Quiescence Signalling in the InProcessPipelineRunner

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397653#comment-15397653
 ] 

ASF GitHub Bot commented on BEAM-156:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/746


> Implement Quiescence Signalling in the InProcessPipelineRunner
> --
>
> Key: BEAM-156
> URL: https://issues.apache.org/jira/browse/BEAM-156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> A pipeline is quiescent when the following two properties hold:
>   There are no triggers that can fire, given the current processing time and 
> watermark
>   All pending elements cannot make progress until a side input produces 
> additional output
> This is approximately equivalent to: If no more input is received, the 
> pipeline will not perform any additional processing absent advances in 
> processing time or event time
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #746: Use AutoValue for StepTransformResult

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/746


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #749: jms: fix artifact ID in pom.xml

2016-07-28 Thread dhalperi

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/749

jms: fix artifact ID in pom.xml



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam jms-fixup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/749.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #749


commit b4e651b4686e498c1d0d529a878e9f6c911356e0
Author: Dan Halperin 
Date:   2016-07-28T06:18:35Z

jms: fix artifact ID in pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Remove unneccessary method in ReadEvaluatorFactories

2016-07-28 Thread lcwik

Remove unneccessary method in ReadEvaluatorFactories

This closes #747


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9c447510
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9c447510
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9c447510

Branch: refs/heads/master
Commit: 9c447510a68abbcdb342073eb7bd1fc179e0bcb7
Parents: a46081e 5637b65
Author: Luke Cwik 
Authored: Thu Jul 28 11:40:40 2016 -0400
Committer: Luke Cwik 
Committed: Thu Jul 28 11:40:40 2016 -0400

--
 .../direct/BoundedReadEvaluatorFactory.java | 15 --
 .../direct/UnboundedReadEvaluatorFactory.java   | 21 ++--
 2 files changed, 10 insertions(+), 26 deletions(-)
--

[1/2] incubator-beam git commit: Remove unneccessary method in ReadEvaluatorFactories

2016-07-28 Thread lcwik

Repository: incubator-beam
Updated Branches:
  refs/heads/master a46081eb5 -> 9c447510a


Remove unneccessary method in ReadEvaluatorFactories

The getTransformEvaluator() method's only call is poll on the result of
getTransformEvaluatorQueue (which is only ever called by
getTransformEvaluator). Instead, move the construction to
getTransformEvaluator and call poll on the result.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5637b654
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5637b654
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5637b654

Branch: refs/heads/master
Commit: 5637b654ea71233dc5a34832d4c44b5f003f36e9
Parents: a46081e
Author: Thomas Groh 
Authored: Fri Jul 22 15:34:03 2016 -0700
Committer: Luke Cwik 
Committed: Thu Jul 28 11:40:10 2016 -0400

--
 .../direct/BoundedReadEvaluatorFactory.java | 15 --
 .../direct/UnboundedReadEvaluatorFactory.java   | 21 ++--
 2 files changed, 10 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5637b654/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 9ba8b61..2f4f86c 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -60,21 +60,14 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 return getTransformEvaluator((AppliedPTransform) application, 
evaluationContext);
   }
 
-  private  TransformEvaluator getTransformEvaluator(
-  final AppliedPTransform, Bounded> 
transform,
-  final EvaluationContext evaluationContext) {
-return getTransformEvaluatorQueue(transform, evaluationContext).poll();
-  }
-
   /**
-   * Get the queue of {@link TransformEvaluator TransformEvaluators} that 
produce elements for the
-   * provided application of {@link Bounded Read.Bounded}, initializing it if 
required.
+   * Get a {@link TransformEvaluator} that produces elements for the provided 
application of
+   * {@link Bounded Read.Bounded}, initializing the queue of evaluators if 
required.
*
* This method is thread-safe, and will only produce new evaluators if no 
other invocation has
* already done so.
*/
-  @SuppressWarnings("unchecked")
-  private  Queue 
getTransformEvaluatorQueue(
+  private  TransformEvaluator getTransformEvaluator(
   final AppliedPTransform, Bounded> 
transform,
   final EvaluationContext evaluationContext) {
 // Key by the application and the context the evaluation is occurring in 
(which call to
@@ -95,7 +88,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 evaluatorQueue = (Queue) 
sourceEvaluators.get(transform);
   }
 }
-return evaluatorQueue;
+return evaluatorQueue.poll();
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5637b654/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index 674be5e..0e2745b 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -35,7 +35,6 @@ import com.google.common.annotations.VisibleForTesting;
 import org.joda.time.Instant;
 
 import java.io.IOException;
-import java.util.Queue;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.ConcurrentMap;
@@ -73,25 +72,17 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 return getTransformEvaluator((AppliedPTransform) application, 
evaluationContext);
   }
 
-  private  TransformEvaluator getTransformEvaluator(
-  final AppliedPTransform, Unbounded> 
transform,
-  final EvaluationContext evaluationContext) {
-return getTransformEvaluatorQueue(transform, evaluationContext).poll();
-  }
-

[2/2] incubator-beam git commit: Closes #726

2016-07-28 Thread robertwb

Closes #726


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/53ab635c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/53ab635c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/53ab635c

Branch: refs/heads/python-sdk
Commit: 53ab635c75e4b94b3930a601911a717ddc499efe
Parents: 26ff657 2f4054b
Author: Robert Bradshaw 
Authored: Thu Jul 28 11:05:56 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:05:56 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Make TextFileReader observable

2016-07-28 Thread robertwb

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 26ff65795 -> 53ab635c7


Make TextFileReader observable

This allows future implementation of size tracking for elements in side input 
sources.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2f4054ba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2f4054ba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2f4054ba

Branch: refs/heads/python-sdk
Commit: 2f4054ba37da1c1100f45a572d96e7a6e2e60152
Parents: 26ff657
Author: Charles Chen 
Authored: Mon Jul 25 11:44:22 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:05:55 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2f4054ba/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 3afaae8..b1e091b 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -745,10 +745,12 @@ class NativeTextFileSink(iobase.NativeSink):
 # TextFileReader, TextMultiFileReader.
 
 
-class TextFileReader(iobase.NativeSourceReader):
+class TextFileReader(iobase.NativeSourceReader,
+ coders.observable.ObservableMixin):
   """A reader for a text file source."""
 
   def __init__(self, source):
+super(TextFileReader, self).__init__()
 self.source = source
 self.start_offset = self.source.start_offset or 0
 self.end_offset = self.source.end_offset
@@ -778,6 +780,7 @@ class TextFileReader(iobase.NativeSourceReader):
   self._file.seek(self.start_offset - 1)
   self.current_offset -= 1
   line = self._file.readline()
+  self.notify_observers(line, is_encoded=True)
   self.current_offset += len(line)
 else:
   self._file.seek(self.start_offset)
@@ -801,6 +804,7 @@ class TextFileReader(iobase.NativeSourceReader):
 # a dynamic split request from the service.
 return
   line = self._file.readline()
+  self.notify_observers(line, is_encoded=True)
   self.current_offset += len(line)
   if self.source.strip_trailing_newlines:
 line = line.rstrip('\n')

[3/3] incubator-beam git commit: Closes #741

2016-07-28 Thread robertwb

Closes #741


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b4716d9d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b4716d9d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b4716d9d

Branch: refs/heads/python-sdk
Commit: b4716d9dcf9c3f198fc72e181adc0bae8b6fa818
Parents: 53ab635 866a09d
Author: Robert Bradshaw 
Authored: Thu Jul 28 11:08:06 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:08:06 2016 -0700

--
 sdks/python/apache_beam/transforms/combiners.py | 201 ---
 .../apache_beam/transforms/combiners_test.py|  42 +++-
 2 files changed, 164 insertions(+), 79 deletions(-)
--

[1/3] incubator-beam git commit: Better top implementation.

2016-07-28 Thread robertwb

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 53ab635c7 -> b4716d9dc


Better top implementation.

When selecting the top k of n, it is common that k << n.
Using a heap is O(n log k) while select algorithms can
achieve O(n + k log k).

This also avoids the ugliness that heapq does not take the
comparator as an argument, resulting in _HeapItem classes that
were cumbersome and expensive to serialize.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/adb3ed93
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/adb3ed93
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/adb3ed93

Branch: refs/heads/python-sdk
Commit: adb3ed93053c83b4e28e7baa879e9aee82f02785
Parents: 53ab635
Author: Robert Bradshaw 
Authored: Wed Jul 27 10:09:49 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:08:05 2016 -0700

--
 sdks/python/apache_beam/transforms/combiners.py | 111 +--
 1 file changed, 51 insertions(+), 60 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/adb3ed93/sdks/python/apache_beam/transforms/combiners.py
--
diff --git a/sdks/python/apache_beam/transforms/combiners.py 
b/sdks/python/apache_beam/transforms/combiners.py
index 8c56e5a..453c0f8 100644
--- a/sdks/python/apache_beam/transforms/combiners.py
+++ b/sdks/python/apache_beam/transforms/combiners.py
@@ -228,76 +228,67 @@ class TopCombineFn(core.CombineFn):
   apply call become additional arguments to the comparator.
   """
 
-  # Actually pickling the comparison operators (including, often, their
-  # entire globals) can be very expensive.  Instead refer to them by index
-  # in this dictionary, which is populated on construction (including
-  # unpickling).
-  compare_by_id = {}
-
-  def __init__(self, n, compare, _compare_id=None):  # pylint: 
disable=invalid-name
+  # TODO(robertwb): Allow taking a key rather than a compare.
+  def __init__(self, n, compare):
 self._n = n
+self._buffer_size = min(2 * n, n + 1000)
 self._compare = compare
-self._compare_id = _compare_id or id(compare)
-TopCombineFn.compare_by_id[self._compare_id] = self._compare
-
-  def __reduce_ex__(self, _):
-return TopCombineFn, (self._n, self._compare, self._compare_id)
 
-  class _HeapItem(object):
-"""A wrapper for values supporting arbitrary comparisons.
-
-The heap implementation supplied by Python is a min heap that always uses
-the __lt__ operator if one is available. This wrapper overloads __lt__,
-letting us specify arbitrary precedence for elements in the PCollection.
-"""
+  def create_accumulator(self, *args, **kwargs):
+return None, []
 
-def __init__(self, item, compare_id, *args, **kwargs):
-  # item: wrapped item.
-  # compare:  an implementation of the pairwise < operator.
-  # args, kwargs: extra arguments supplied to the compare function.
-  self.item = item
-  self.compare_id = compare_id
-  self.args = args
-  self.kwargs = kwargs
+  def add_input(self, accumulator, element, *args, **kwargs):
+if args or kwargs:
+  lt = lambda a, b: self._compare(a, b, *args, **kwargs)
+else:
+  lt = self._compare
 
-def __lt__(self, other):
-  return TopCombineFn.compare_by_id[self.compare_id](
-  self.item, other.item, *self.args, **self.kwargs)
+threshold, buffer = accumulator
+if len(buffer) < self._n:
+  if not buffer:
+return element, [element]
+  else:
+buffer.append(element)
+if lt(element, threshold):  # element < threshold
+  return element, buffer
+else:
+  return accumulator  # with mutated buffer
+elif lt(threshold, element):  # threshold < element
+  buffer.append(element)
+  if len(buffer) < self._buffer_size:
+return accumulator
+  else:
+buffer.sort(cmp=lambda a, b: (not lt(a, b)) - (not lt(b, a)))
+return buffer[-self._n], buffer[-self._n:]
+else:
+  return accumulator
 
-  def create_accumulator(self, *args, **kwargs):
-return []  # Empty heap.
-
-  def add_input(self, heap, element, *args, **kwargs):
-# Note that because heap is a min heap, heappushpop will discard incoming
-# elements that are lesser (according to compare) than those in the heap
-# (since that's what you would get if you pushed a small element on and
-# popped the smallest element off). So, filtering a collection with a
-# min-heap gives you the largest elements in the collection.
-item = self._HeapItem(element, self._compare_id, *args, **kwargs)
-if len(heap) < self._n:
-  heapq.heappush(heap, item)

[1/2] incubator-beam git commit: Optimize Map and Flatmap when there are no side inputs.

2016-07-28 Thread robertwb

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk b4716d9dc -> 351c3831d


Optimize Map and Flatmap when there are no side inputs.

varargs and kwargs are expensive, even when they're empty.

This is especially true for otherwise one-argument Python calls
which are special cased in CPython.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7d2fb1f8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7d2fb1f8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7d2fb1f8

Branch: refs/heads/python-sdk
Commit: 7d2fb1f88d1a2370dd4053f3a1738cbb9838cc2f
Parents: b4716d9
Author: Robert Bradshaw 
Authored: Wed Jul 27 18:29:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:09:48 2016 -0700

--
 sdks/python/apache_beam/transforms/core.py | 40 ++---
 1 file changed, 30 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7d2fb1f8/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 5e6aafc..38b9cd2 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -20,6 +20,8 @@
 from __future__ import absolute_import
 
 import copy
+import inspect
+import types
 
 from apache_beam import pvalue
 from apache_beam import typehints
@@ -194,6 +196,16 @@ class DoFn(WithTypeHints):
   return type_hint
 
 
+def _fn_takes_side_inputs(fn):
+  try:
+argspec = inspect.getargspec(fn)
+  except TypeError:
+# We can't tell; maybe it does.
+return True
+  is_bound = isinstance(fn, types.MethodType) and fn.im_self is not None
+  return len(argspec.args) > 1 + is_bound or argspec.varargs or 
argspec.keywords
+
+
 class CallableWrapperDoFn(DoFn):
   """A DoFn (function) object wrapping a callable object.
 
@@ -214,6 +226,11 @@ class CallableWrapperDoFn(DoFn):
   raise TypeError('Expected a callable object instead of: %r' % fn)
 
 self._fn = fn
+if _fn_takes_side_inputs(fn):
+  self.process = lambda context, *args, **kwargs: fn(
+  context.element, *args, **kwargs)
+else:
+  self.process = lambda context: fn(context.element)
 
 super(CallableWrapperDoFn, self).__init__()
 
@@ -237,9 +254,6 @@ class CallableWrapperDoFn(DoFn):
 return self._strip_output_annotations(
 trivial_inference.infer_return_type(self._fn, [input_type]))
 
-  def process(self, context, *args, **kwargs):
-return self._fn(context.element, *args, **kwargs)
-
   def process_argspec_fn(self):
 return getattr(self._fn, '_argspec_fn', self._fn)
 
@@ -676,7 +690,10 @@ def Map(fn_or_label, *args, **kwargs):  # pylint: 
disable=invalid-name
 'Map can be used only with callable objects. '
 'Received %r instead for %s argument.'
 % (fn, 'first' if label is None else 'second'))
-  wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
+  if _fn_takes_side_inputs(fn):
+wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
+  else:
+wrapper = lambda x: [fn(x)]
 
   # Proxy the type-hint information from the original function to this new
   # wrapped function.
@@ -1008,21 +1025,24 @@ class GroupByKey(PTransform):
   value_type = windowed_value_iter_type.inner_type.inner_type
   return Iterable[KV[key_type, Iterable[value_type]]]
 
-def process(self, context):
-  k, vs = context.element
+def start_bundle(self, context):
   # pylint: disable=wrong-import-order, wrong-import-position
   from apache_beam.transforms.trigger import InMemoryUnmergedState
   from apache_beam.transforms.trigger import create_trigger_driver
   # pylint: enable=wrong-import-order, wrong-import-position
-  driver = create_trigger_driver(self.windowing, True)
-  state = InMemoryUnmergedState()
+  self.driver = create_trigger_driver(self.windowing, True)
+  self.state_type = InMemoryUnmergedState
+
+def process(self, context):
+  k, vs = context.element
+  state = self.state_type()
   # TODO(robertwb): Conditionally process in smaller chunks.
-  for wvalue in driver.process_elements(state, vs, MIN_TIMESTAMP):
+  for wvalue in self.driver.process_elements(state, vs, MIN_TIMESTAMP):
 yield wvalue.with_value((k, wvalue.value))
   while state.timers:
 fired = state.get_and_clear_timers()
 for timer_window, (name, time_domain, fire_time) in fired:
-  for wvalue in driver.process_timer(
+  for wvalue in self.driver.process_timer(
   timer_window, name, time_domain, fire_time,

[GitHub] incubator-beam pull request #748: Optimize Map and Flatmap when there are no...

2016-07-28 Thread robertwb

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/748


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #741: Top improvements

2016-07-28 Thread robertwb

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/741


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-256) Add lifecycle event verifiers for Beam pipelines.

2016-07-28 Thread Jason Kuster (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Kuster resolved BEAM-256.
---
   Resolution: Fixed
Fix Version/s: 0.1.0-incubating

> Add lifecycle event verifiers for Beam pipelines.
> -
>
> Key: BEAM-256
> URL: https://issues.apache.org/jira/browse/BEAM-256
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
> Fix For: 0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #748

2016-07-28 Thread robertwb

Closes #748


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/351c3831
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/351c3831
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/351c3831

Branch: refs/heads/python-sdk
Commit: 351c3831de1bdcfcb19b2f24f9f0b6a19e77e421
Parents: b4716d9 7d2fb1f
Author: Robert Bradshaw 
Authored: Thu Jul 28 11:09:49 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 11:09:49 2016 -0700

--
 sdks/python/apache_beam/transforms/core.py | 40 ++---
 1 file changed, 30 insertions(+), 10 deletions(-)
--

[jira] [Assigned] (BEAM-496) Remove the notion of a Fake subscription from PubsubIO

2016-07-28 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-496:
--

Assignee: Daniel Halperin  (was: Davor Bonaci)

> Remove the notion of a Fake subscription from PubsubIO
> --
>
> Key: BEAM-496
> URL: https://issues.apache.org/jira/browse/BEAM-496
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> PubsubSubscription should be cleaned up and references to a "Fake" type 
> removed.
> start_signal/ is no longer needed since StreamingCreate has been replaced 
> with an UnboundedReadFromBoundedSource.
> random/ was created but never used in Dataflow or Apache Beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #753: [BEAM-436] Re-enable RunnableOnService tes...

2016-07-28 Thread swegner

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/753

[BEAM-436] Re-enable RunnableOnService tests which need tempRoot

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

@peihe's recent work to improve tempRoot handling means that these tests 
are now working properly. They can be re-enabled to validate runner display 
data for i/o transforms.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam io-runnableonservice

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/753.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #753


commit e741f32184838f598d351117e024995ff4f5d3fe
Author: Scott Wegner 
Date:   2016-07-28T20:35:55Z

Re-enable RunnableOnService tests which need tempRoot




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/4] incubator-beam-site git commit: Merged in pull#25 and regenerated html.

2016-07-28 Thread frances

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 5d7ef1d64 -> ea05f978a


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/docs/index.md
--
diff --git a/docs/index.md b/docs/index.md
index e825ed3..c2523df 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,27 +1,26 @@
 ---
 layout: default
 ---
-
-
-
-The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
-
+
+  
+  
+  
+  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
+  
+
 
-# Apache Beam Documentation
+# Beam Documentation
 
-Welcome to the Apache Beam project documentation! The following resources can 
help you use, integrate with, and contribute to Apache Beam.
+Welcome to the Beam documentation! Here you'll find resources to help you use, 
learn about, and contribute to Beam.
 
-## Using Beam
+### [Use](/docs/use/index.html)
 
-These resources help you build and run Beam pipelines for your data processing 
tasks.
+Learn how to use Beam to create data processing pipelines.
 
-* [Beam Programming Guide](/coming-soon.html) - Learn how to use the Beam SDKs 
to define your data processing workflows.
-* [Beam Example Walkthroughs](/coming-soon.html) - Check out detailed 
walkthroughs of runnable Beam pipelines that use the Beam SDKs to tackle a 
variety of use cases.
+### [Learn](/docs/learn/index.html)
 
-## Integrating your Distribured Processing Back-end with Beam
+Find conceptual information on the Beam programming model as well as detailed 
reference information on the Beam SDKs and Pipeline Runners.
 
-* [Beam Runner Developer's Guide](/coming-soon.html) - Learn how to build a 
Beam Runner to help Beam pipelines work with your distributed processing 
back-end.
+### [Contribute](/docs/contribute/index.html)
 
-## Contributing to Beam
-
-* [Beam Contribution Guide](/contribution-guide/) - Learn how to contribute to 
the various open-source Beam SDKs.
+Discover how you can contribute to the Beam project.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/docs/learn/index.md
--
diff --git a/docs/learn/index.md b/docs/learn/index.md
new file mode 100644
index 000..2d78762
--- /dev/null
+++ b/docs/learn/index.md
@@ -0,0 +1,51 @@
+---
+layout: default
+---
+
+  
+  
+  
+  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
+  
+
+
+# Learn about the Beam Model
+
+Get in-depth conceptual information and reference material for the Beam SDKs 
and Runners:
+
+ [Beam Programming Guide]({{ site.baseurl 
}}/docs/learn/programming-guide/) 
+Learn about the Beam Programming Model and the concepts common to all Beam 
SDKs and Runners.
+
+ Beam SDKs
+Find status and reference information on all of the available Beam SDKs.
+
+* SDK for Java
+  * Status
+  * Supported I/O Connectors
+  * API Reference
+* SDK for Python
+  * Status
+  * Supported I/O Connectors
+  * API Reference
+
+ Pipeline Runners
+Find status and reference information on all of the available Beam Pipeline 
Runners.
+
+* [Capability Matrix]({{ site.baseurl }}/capability-matrix/)
+* Spark Runner
+  * Status
+  * Pipeline Options for Spark
+* Flink Runner
+  * Status
+  * Pipeline Options for Flink
+* Google Cloud Dataflow
+  * Status
+  * Pipeline Options for Google Cloud Dataflow
+
+ Additional Resources
+More information to help you learn about Beam.
+
+* [Additional Technical Documentation](https://goo.gl/ps8twC)
+* Tech Talks
+* Dataflow Paper
+* Streaming 101/102
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/docs/learn/programming-guide.md
--
diff --git a/docs/learn/programming-guide.md b/docs/learn/programming-guide.md
new file mode 100644
index 000..7e4afb5
--- /dev/null
+++ b/docs/learn/programming-guide.md
@@ -0,0 +1,416 @@
+---
+layout: default
+---
+
+  
+  
+  
+  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For

[2/4] incubator-beam-site git commit: Merged in pull#25 and regenerated html.

2016-07-28 Thread frances

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/content/docs/use/beam-overview.html
--
diff --git a/content/docs/use/beam-overview.html 
b/content/docs/use/beam-overview.html
new file mode 100644
index 000..9c798e3
--- /dev/null
+++ b/content/docs/use/beam-overview.html
@@ -0,0 +1,196 @@
+
+
+
+  
+  
+  
+  
+
+  Apache Beam (incubating)
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/docs/use/beam-overview.html;>
+  http://beam.incubator.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+ Use 
+ 
+ User Hub
+ 
+ Beam 
Overview
+ Quickstart
+ Example Walkthroughs
+ WordCount
+ Mobile 
Gaming
+ 
+ Release Information
+ Support Resources
+ Mailing Lists
+  Issue Tracking
+ http://stackoverflow.com/questions/tagged/apache-beam;>Beam on 
StackOverflow
+  http://apachebeam.slack.com;>Beam Slack 
Channel
+ 
+   
+
+ Learn 
+ 
+ Learner Hub
+ 
+ Beam 
Programming Guide
+ Capability 
Matrix
+ https://goo.gl/ps8twC;>Additional 
Technical Docs
+ 
+   
+
+ Contribute 
+ 
+ Contributor 
Hub
+ 
+ Contribution 
Guide
+ https://goo.gl/nk5OM0;>Technical 
Vision
+ Resources
+ Mailing Lists
+  Source Repository
+  Issue Tracking
+ Project Materials
+ Presentation 
Materials
+ Public 
Meetings
+ 
+ Apache Beam Team
+ 
+   
+Blog
+
+  Project 
+  
+Presentation 
Materials
+Logos and design
+http://apache.org/licenses/LICENSE-2.0.html;>License
+  
+
+  
+
+  
+
+
+
+
+
+
+
+
+  
+Apache Beam Overview
+
+Apache Beam is an open source, unified programming model that you can use 
to create a data processing pipeline. You start by building a 
program that defines the pipeline using one of the open source Beam SDKs. The 
pipeline is then executed by one of Beamâs supported distributed 
processing back-ends, which include http://flink.apache.org;>Apache Flink, http://spark.apache.org;>Apache Spark, and https://cloud.google.com/dataflow;>Google Cloud Dataflow.
+
+Beam is particularly useful for http://en.wikipedia.org/wiki/Embarassingly_parallel;>Embarrassingly 
Parallel data processing tasks, in which the problem can be decomposed into 
many smaller bundles of data that can be processed independently and in 
parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks 
and pure data integration. These tasks are useful for moving data between 
different storage media and data sources, transforming data into a more 
desirable format, or loading data onto a new system.
+
+Apache Beam SDKs
+
+The Beam SDKs provide a unified programming model that can represent and 
transform data sets of any size, whether the input is a finite data set from a 
batch data source, or an infinite data set from a streaming data source. The 
Beam SDKs use the same classes to represent both bounded and unbounded data, 
and the same transforms to operate on that data. You use the Beam SDK of your 
choice to build a program that defines your data processing pipeline.
+
+Beam currently supports the following language-specific SDKs:
+
+
+
+  Language
+  SDK Status
+
+
+  Java
+  Active Development
+
+
+  Python
+  Coming Soon
+
+
+  Other
+  TBD
+
+
+
+Apache Beam Pipeline Runners
+
+The Beam Pipeline Runners translate the data processing pipeline you define 
with your Beam program into the API compatible with the distributed

[jira] [Created] (BEAM-496) Remove the notion of a Fake subscription from PubsubIO

2016-07-28 Thread Luke Cwik (JIRA)

Luke Cwik created BEAM-496:
--

 Summary: Remove the notion of a Fake subscription from PubsubIO
 Key: BEAM-496
 URL: https://issues.apache.org/jira/browse/BEAM-496
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core, sdk-java-gcp
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


PubsubSubscription should be cleaned up and references to a "Fake" type removed.

start_signal/ is no longer needed since StreamingCreate has been replaced with 
an UnboundedReadFromBoundedSource.
random/ was created but never used in Dataflow or Apache Beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-497) Modify offset range tracker to use first response as start offset

2016-07-28 Thread Chamikara Jayalath (JIRA)

Chamikara Jayalath created BEAM-497:
---

 Summary: Modify offset range tracker to use first response as 
start offset
 Key: BEAM-497
 URL: https://issues.apache.org/jira/browse/BEAM-497
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Update Python SDK to match https://github.com/apache/incubator-beam/pull/515



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #25: Creates preliminary landing pages for ...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/25


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] incubator-beam-site git commit: Merged in pull#25 and regenerated html.

2016-07-28 Thread frances

Merged in pull#25 and regenerated html.

This closes #25.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ea05f978
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ea05f978
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ea05f978

Branch: refs/heads/asf-site
Commit: ea05f978afb90d41868e8275e7accac96765f2bd
Parents: 5d7ef1d
Author: Frances Perry 
Authored: Thu Jul 28 17:03:42 2016 -0700
Committer: Frances Perry 
Committed: Thu Jul 28 17:03:42 2016 -0700

--
 _includes/header.html   |  73 ++-
 .../2016/03/17/capability-matrix.html   |  73 ++-
 .../2016/04/03/presentation-materials.html  |  73 ++-
 .../sdk/2016/02/25/python-sdk-now-public.html   |  73 ++-
 .../beam/release/2016/06/15/first-release.html  |  73 ++-
 .../website/2016/02/22/beam-has-a-logo.html |  73 ++-
 .../blog/2016/05/18/splitAtFraction-method.html |  73 ++-
 .../05/27/where-is-my-pcollection-dot-map.html  |  73 ++-
 .../06/13/flink-batch-runner-milestone.html |  73 ++-
 content/blog/index.html |  73 ++-
 content/capability-matrix/index.html|  75 ++-
 content/coming-soon.html|  73 ++-
 content/contribution-guide/index.html   |  77 ++-
 content/docs/contribute/index.html  | 183 ++
 content/docs/index.html | 110 ++--
 content/docs/learn/index.html   | 208 +++
 content/docs/learn/programming-guide.html   | 597 +++
 content/docs/use/beam-overview.html | 196 ++
 content/docs/use/index.html | 176 ++
 content/docs/use/walkthroughs.html  | 150 +
 content/feed.xml|   6 +-
 content/getting_started/index.html  |  73 ++-
 content/index.html  |  73 ++-
 content/issue_tracking/index.html   |  73 ++-
 content/mailing_lists/index.html|  73 ++-
 content/material/index.html |  73 ++-
 content/presentation-materials/index.html   |  73 ++-
 content/privacy_policy/index.html   |  73 ++-
 content/public-meetings/index.html  |  73 ++-
 content/releases/index.html |  73 ++-
 content/source_repository/index.html|  73 ++-
 content/team/index.html |  73 ++-
 docs/contribute/index.md|  41 ++
 docs/index.md   |  31 +-
 docs/learn/index.md |  51 ++
 docs/learn/programming-guide.md | 416 +
 docs/use/beam-overview.md   |  60 ++
 docs/use/index.md   |  36 ++
 docs/use/walkthroughs.md|  13 +
 39 files changed, 3288 insertions(+), 744 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index d66803a..9c98db9 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -14,35 +14,52 @@
 
   
 
-  Documentation 
-  
-Releases
-Release 
information
-
-Guides
-Getting 
Started
-
-Technical Documentation
-Java 
API Reference
-Capability 
Matrix
-https://goo.gl/ps8twC;>Technical Docs
-https://goo.gl/nk5OM0;>Technical Vision
-  
-
+ Use 
+ 
+ User 
Hub
+ 
+ Beam Overview
+ Quickstart
+ Example Walkthroughs
+ WordCount
+ Mobile Gaming
+ 
+ Release 
Information
+ Support Resources
+ Mailing Lists
+  Issue 
Tracking
+ http://stackoverflow.com/questions/tagged/apache-beam;>Beam on 
StackOverflow
+  http://apachebeam.slack.com;>Beam Slack 
Channel
+ 
+   
 
-  Community 
-  
-Community
-Mailing 
Lists
-Apache Beam Team
-Public 
Meetings
-
-Contribute
-Contribution 
Guide
-Source 
Repository
-Issue 
Tracking
-  
-
+ Learn 
+ 
+

[3/4] incubator-beam-site git commit: Merged in pull#25 and regenerated html.

2016-07-28 Thread frances

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/content/docs/index.html
--
diff --git a/content/docs/index.html b/content/docs/index.html
index 7327bed..71fe449 100644
--- a/content/docs/index.html
+++ b/content/docs/index.html
@@ -48,35 +48,52 @@
 
   
 
-  Documentation 
-  
-Releases
-Release information
-
-Guides
-Getting Started
-
-Technical Documentation
-Java API 
Reference
-Capability Matrix
-https://goo.gl/ps8twC;>Technical Docs
-https://goo.gl/nk5OM0;>Technical Vision
-  
-
+ Use 
+ 
+ User Hub
+ 
+ Beam 
Overview
+ Quickstart
+ Example Walkthroughs
+ WordCount
+ Mobile 
Gaming
+ 
+ Release Information
+ Support Resources
+ Mailing Lists
+  Issue Tracking
+ http://stackoverflow.com/questions/tagged/apache-beam;>Beam on 
StackOverflow
+  http://apachebeam.slack.com;>Beam Slack 
Channel
+ 
+   
 
-  Community 
-  
-Community
-Mailing Lists
-Apache Beam Team
-Public Meetings
-
-Contribute
-Contribution Guide
-Source Repository
-Issue Tracking
-  
-
+ Learn 
+ 
+ Learner Hub
+ 
+ Beam 
Programming Guide
+ Capability 
Matrix
+ https://goo.gl/ps8twC;>Additional 
Technical Docs
+ 
+   
+
+ Contribute 
+ 
+ Contributor 
Hub
+ 
+ Contribution 
Guide
+ https://goo.gl/nk5OM0;>Technical 
Vision
+ Resources
+ Mailing Lists
+  Source Repository
+  Issue Tracking
+ Project Materials
+ Presentation 
Materials
+ Public 
Meetings
+ 
+ Apache Beam Team
+ 
+   
 Blog
 
   Project 
@@ -98,36 +115,29 @@
 
 
   
-
-
-
-The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
-
-
-Apache Beam Documentation
+
+  
+  
+  
+  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
+  
+
 
-Welcome to the Apache Beam project documentation! The following resources 
can help you use, integrate with, and contribute to Apache Beam.
+Beam Documentation
 
-Using Beam
+Welcome to the Beam documentation! Here youâll find resources to help you 
use, learn about, and contribute to Beam.
 
-These resources help you build and run Beam pipelines for your data 
processing tasks.
+Use
 
-
-  Beam Programming Guide - Learn how to 
use the Beam SDKs to define your data processing workflows.
-  Beam Example Walkthroughs - Check out 
detailed walkthroughs of runnable Beam pipelines that use the Beam SDKs to 
tackle a variety of use cases.
-
+Learn how to use Beam to create data processing pipelines.
 
-Integrating 
your Distribured Processing Back-end with Beam
+Learn
 
-
-  Beam Runner Developerâs Guide - Learn 
how to build a Beam Runner to help Beam pipelines work with your distributed 
processing back-end.
-
+Find conceptual information on the Beam programming model as well as 
detailed reference information on the Beam SDKs and Pipeline Runners.
 
-Contributing to Beam
+Contribute
 
-
-  Beam Contribution Guide - Learn how 
to contribute to the various open-source Beam SDKs.
-
+Discover how you can contribute to the Beam project.
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ea05f978/content/docs/learn/index.html
--
diff --git a/content/docs/learn/index.html b/content/docs/learn/index.html
new file mode 100644
index

[1/2] incubator-beam git commit: Increased the GCS buffer size from 1MB to 8MB and introduced a 128kB buffer for the pipe.

2016-07-28 Thread robertwb

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 351c3831d -> c155ef0eb


Increased the GCS buffer size from 1MB to 8MB and introduced a 128kB buffer for 
the pipe.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a1f1fa06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a1f1fa06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a1f1fa06

Branch: refs/heads/python-sdk
Commit: a1f1fa06ee8683273182548e7eb2d6612040d2bf
Parents: 351c383
Author: Marian Dvorsky 
Authored: Thu Jul 28 13:02:15 2016 -0700
Committer: Marian Dvorsky 
Committed: Thu Jul 28 13:02:15 2016 -0700

--
 sdks/python/apache_beam/io/gcsio.py | 30 +-
 1 file changed, 21 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a1f1fa06/sdks/python/apache_beam/io/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcsio.py 
b/sdks/python/apache_beam/io/gcsio.py
index 9377266..88fcfb8 100644
--- a/sdks/python/apache_beam/io/gcsio.py
+++ b/sdks/python/apache_beam/io/gcsio.py
@@ -49,6 +49,7 @@ except ImportError:
 
 
 DEFAULT_READ_BUFFER_SIZE = 1024 * 1024
+WRITE_CHUNK_SIZE = 8 * 1024 * 1024
 
 
 def parse_gcs_path(gcs_path):
@@ -546,6 +547,10 @@ class GcsBufferedWriter(object):
 self.closed = False
 self.position = 0
 
+# A small buffer to avoid CPU-heavy per-write pipe calls.
+self.write_buffer = bytearray()
+self.write_buffer_size = 128 * 1024
+
 # Set up communication with uploading thread.
 parent_conn, child_conn = multiprocessing.Pipe()
 self.child_conn = child_conn
@@ -557,7 +562,7 @@ class GcsBufferedWriter(object):
 bucket=self.bucket,
 name=self.name))
 self.upload = transfer.Upload(GcsBufferedWriter.PipeStream(child_conn),
-  mime_type)
+  mime_type, chunksize=WRITE_CHUNK_SIZE)
 self.upload.strategy = transfer.RESUMABLE_UPLOAD
 
 # Start uploading thread.
@@ -598,14 +603,10 @@ class GcsBufferedWriter(object):
 self._check_open()
 if not data:
   return
-try:
-  self.conn.send_bytes(data)
-  self.position += len(data)
-except IOError:
-  if self.upload_thread.last_error:
-raise self.upload_thread.last_error  # pylint: disable=raising-bad-type
-  else:
-raise
+self.write_buffer.extend(data)
+if len(self.write_buffer) > self.write_buffer_size:
+  self._flush_write_buffer()
+self.position += len(data)
 
   def tell(self):
 """Return the total number of bytes passed to write() so far."""
@@ -613,6 +614,7 @@ class GcsBufferedWriter(object):
 
   def close(self):
 """Close the current GCS file."""
+self._flush_write_buffer()
 self.closed = True
 self.conn.close()
 self.upload_thread.join()
@@ -635,3 +637,13 @@ class GcsBufferedWriter(object):
 
   def writable(self):
 return True
+
+  def _flush_write_buffer(self):
+try:
+  self.conn.send_bytes(buffer(self.write_buffer))
+  self.write_buffer = bytearray()
+except IOError:
+  if self.upload_thread.last_error:
+raise self.upload_thread.last_error  # pylint: disable=raising-bad-type
+  else:
+raise

[GitHub] incubator-beam pull request #752: Beam Python GCSIO write performance improv...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/752


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #752

2016-07-28 Thread robertwb

Closes #752


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c155ef0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c155ef0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c155ef0e

Branch: refs/heads/python-sdk
Commit: c155ef0ebc88ac34a3681b8bff6152e1857da847
Parents: 351c383 a1f1fa0
Author: Robert Bradshaw 
Authored: Thu Jul 28 18:17:43 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 28 18:17:43 2016 -0700

--
 sdks/python/apache_beam/io/gcsio.py | 30 +-
 1 file changed, 21 insertions(+), 9 deletions(-)
--

[2/2] incubator-beam git commit: [BEAM-443] Add waitToFinish() and cancel() in PipelineResult.

2016-07-28 Thread dhalperi

[BEAM-443] Add waitToFinish() and cancel() in PipelineResult.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1b4b7c76
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1b4b7c76
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1b4b7c76

Branch: refs/heads/master
Commit: 1b4b7c762825bba90b80b4548bc442ae9c813dca
Parents: 1df6f5f
Author: Pei He 
Authored: Mon Jul 11 19:42:02 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 20:00:27 2016 -0700

--
 .../beam/examples/common/ExampleUtils.java  |  4 +-
 .../beam/runners/direct/DirectRunner.java   | 18 +
 .../beam/runners/flink/FlinkRunnerResult.java   | 22 -
 .../dataflow/BlockingDataflowRunner.java|  5 +-
 .../runners/dataflow/DataflowPipelineJob.java   | 85 +++-
 .../beam/runners/dataflow/DataflowRunner.java   |  5 +-
 .../dataflow/testing/TestDataflowRunner.java|  7 +-
 .../dataflow/BlockingDataflowRunnerTest.java|  9 +--
 .../dataflow/DataflowPipelineJobTest.java   | 43 ++
 .../testing/TestDataflowRunnerTest.java | 34 
 .../spark/translation/EvaluationContext.java| 21 +
 .../org/apache/beam/sdk/PipelineResult.java | 38 +
 .../main/java/common/DataflowExampleUtils.java  |  5 +-
 13 files changed, 202 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b4b7c76/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
index 8f9be31..8b66861 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
@@ -306,12 +306,12 @@ public class ExampleUtils {
 addShutdownHook(jobsToCancel);
   }
   try {
-job.waitToFinish(-1, TimeUnit.SECONDS, new 
MonitoringUtil.LoggingHandler());
+job.waitUntilFinish();
   } catch (Exception e) {
 throw new RuntimeException("Failed to wait for job to finish: " + 
job.getJobId());
   }
 } else {
-  // Do nothing if the given PipelineResult doesn't support waitToFinish(),
+  // Do nothing if the given PipelineResult doesn't support 
waitUntilFinish(),
   // such as EvaluationResults returned by DirectRunner.
   tearDown();
   printPendingMessages();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b4b7c76/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 72194da..743c565 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -51,8 +51,10 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 
+import org.joda.time.Duration;
 import org.joda.time.Instant;
 
+import java.io.IOException;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.Map;
@@ -362,5 +364,21 @@ public class DirectRunner
   }
   return state;
 }
+
+@Override
+public State cancel() throws IOException {
+  throw new UnsupportedOperationException("DirectPipelineResult does not 
support cancel.");
+}
+
+@Override
+public State waitUntilFinish() throws IOException {
+  return waitUntilFinish(Duration.millis(-1));
+}
+
+@Override
+public State waitUntilFinish(Duration duration) throws IOException {
+  throw new UnsupportedOperationException(
+  "DirectPipelineResult does not support waitUntilFinish.");
+}
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b4b7c76/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
index a8f4cac..cae0b2a 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
+++

[jira] [Commented] (BEAM-443) PipelineResult needs waitToFinish() and cancel()

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398635#comment-15398635
 ] 

ASF GitHub Bot commented on BEAM-443:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/642


> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-443
> URL: https://issues.apache.org/jira/browse/BEAM-443
> Project: Beam
>  Issue Type: New Feature
>Reporter: Pei He
>Assignee: Pei He
>
> waitToFinish() and cancel() are two most common operations for users to 
> interact with a started pipeline.
> Right now, they are only available in DataflowPipelineJob. But, it is better 
> to move them to the common interface, so people can start implement them in 
> other runners, and runner agnostic code can interact with PipelineResult 
> better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #642

2016-07-28 Thread dhalperi

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1df6f5f97 -> 119da4a82


Closes #642


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/119da4a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/119da4a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/119da4a8

Branch: refs/heads/master
Commit: 119da4a82611662486f57536168443277c926790
Parents: 1df6f5f 1b4b7c7
Author: Dan Halperin 
Authored: Thu Jul 28 20:00:27 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 28 20:00:27 2016 -0700

--
 .../beam/examples/common/ExampleUtils.java  |  4 +-
 .../beam/runners/direct/DirectRunner.java   | 18 +
 .../beam/runners/flink/FlinkRunnerResult.java   | 22 -
 .../dataflow/BlockingDataflowRunner.java|  5 +-
 .../runners/dataflow/DataflowPipelineJob.java   | 85 +++-
 .../beam/runners/dataflow/DataflowRunner.java   |  5 +-
 .../dataflow/testing/TestDataflowRunner.java|  7 +-
 .../dataflow/BlockingDataflowRunnerTest.java|  9 +--
 .../dataflow/DataflowPipelineJobTest.java   | 43 ++
 .../testing/TestDataflowRunnerTest.java | 34 
 .../spark/translation/EvaluationContext.java| 21 +
 .../org/apache/beam/sdk/PipelineResult.java | 38 +
 .../main/java/common/DataflowExampleUtils.java  |  5 +-
 13 files changed, 202 insertions(+), 94 deletions(-)
--

[GitHub] incubator-beam pull request #642: [BEAM-443] Promote waitToFinish() and canc...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/642


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #710

2016-07-28 Thread kenn

This closes #710


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/36720a62
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/36720a62
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/36720a62

Branch: refs/heads/master
Commit: 36720a62db2c775a40f220d96b1ea83978ea3259
Parents: 119da4a ea8936b
Author: Kenneth Knowles 
Authored: Thu Jul 28 21:10:59 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 28 21:10:59 2016 -0700

--
 runners/flink/runner/pom.xml | 114 +-
 1 file changed, 63 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/36720a62/runners/flink/runner/pom.xml
--

[1/2] incubator-beam git commit: Move local Flink integration tests to a profile

2016-07-28 Thread kenn

Repository: incubator-beam
Updated Branches:
  refs/heads/master 119da4a82 -> 36720a62d


Move local Flink integration tests to a profile

These tests can now be activated on the commandline
with -P local-runnable-on-service-tests and are
activated directly by Jenkins.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ea8936b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ea8936b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ea8936b3

Branch: refs/heads/master
Commit: ea8936b3365450ae323ce8358cc7645cdced54ad
Parents: 07a313f
Author: Kenneth Knowles 
Authored: Thu Jul 21 14:54:17 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 28 21:10:43 2016 -0700

--
 runners/flink/runner/pom.xml | 114 +-
 1 file changed, 63 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ea8936b3/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 1c522fc..ec3f501 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -32,6 +32,69 @@
 
   jar
 
+  
+
+  local-runnable-on-service-tests
+  false
+  
+
+  
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+  
+  
+runnable-on-service-tests
+integration-test
+
+  test
+
+
+  
+
+  [
+"--runner=TestFlinkRunner",
+"--streaming=false"
+  ]
+
+  
+
+  
+
+  
+  
+streaming-runnable-on-service-tests
+integration-test
+
+  test
+
+
+  true
+  
org.apache.beam.sdk.testing.RunnableOnService
+  none
+  true
+  
+
org.apache.beam:beam-sdks-java-core
+  
+  
+
+  [
+"--runner=TestFlinkRunner",
+"--streaming=true"
+  ]
+
+  
+
+  
+
+  
+
+  
+
+  
+
   
 
 
@@ -191,57 +254,6 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-
-  
-runnable-on-service-tests
-integration-test
-
-  test
-
-
-  org.apache.beam.sdk.testing.RunnableOnService
-  none
-  true
-  
-org.apache.beam:beam-sdks-java-core
-  
-  
-
-  [
-"--runner=TestFlinkRunner",
-"--streaming=false"
-  ]
-
-  
-
-  
-  
-streaming-runnable-on-service-tests
-integration-test
-
-  test
-
-
-  true
-  org.apache.beam.sdk.testing.RunnableOnService
-  none
-  true
-  
-org.apache.beam:beam-sdks-java-core
-  
-  
-
-  [
-"--runner=TestFlinkRunner",
-"--streaming=true"
-  ]
-
-  
-  
-  
-
-  
-

[GitHub] incubator-beam pull request #710: [BEAM-479] Move local Flink integration te...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/710


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-479) Move RunnableOnService test executions to postcommit

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398678#comment-15398678
 ] 

ASF GitHub Bot commented on BEAM-479:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/710


> Move RunnableOnService test executions to postcommit
> 
>
> Key: BEAM-479
> URL: https://issues.apache.org/jira/browse/BEAM-479
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-flink, runner-gearpump, 
> runner-spark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The Spark and Flink RunnableOnService tests only use a local endpoint, so 
> they _can_ run as unit tests, but our test duration is getting out of hand. 
> For Gearpump, the tests timeout. So this ticket tracks getting everyone to a 
> symmetric configuration.
> Later, we can re-enable a select few local endpoint tests for the various 
> runners, to smoke test, and hopefully get actual cluster-based integration 
> tests running on Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #852

2016-07-28 Thread Apache Jenkins Server

See

[GitHub] incubator-beam pull request #755: Added Bigtable Read DisplayData

2016-07-28 Thread ianzhou1

GitHub user ianzhou1 opened a pull request:

https://github.com/apache/incubator-beam/pull/755

Added Bigtable Read DisplayData

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ianzhou1/incubator-beam BigtableDisplayData

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/755.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #755


commit 220d85b93e76571b3fe3c9fa0ec84a811fc59e9a
Author: Ian Zhou 
Date:   2016-07-29T00:46:21Z

Added Bigtable Read DisplayData




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #756: Replace ParDo with MapElements and FlatMap...

2016-07-28 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/756

Replace ParDo with MapElements and FlatMapElements where possible

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The commits ended up having fairly separate topics, but can be reviewed 
individually or as a medium-sized change.

1. The first commit replaces `ParDo` with `MapElements` and 
`FlatMapElements` where it is easy to do so.
2. While debugging, I noticed that `DoFn` used a less-powerful form of 
`TypeDescriptor` and switched trivially to the enhanced version.
3. The root cause of issues with `MapElements` and `FlatMapElements` was a 
lack of use of the input type descriptor. Making it available involved a 
moderate refactor. In the process I broke some tests to do with display data 
and fixed them plus enhancements to display data for `SimpleFunction`.

If reviewers insist, I can try to alter this commit history.

R: @bjchambers AND @swegner 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam map-flatmap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/756.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #756


commit b041197382f6a4ea5f6ad93f5e6f32aa1212937f
Author: Kenneth Knowles 
Date:   2016-07-27T21:23:15Z

Replace ParDo with simpler transforms where possible

There are a number of places in the Java SDK where we use
ParDo.of(DoFn) when MapElements or other higher-level
composites are applicable and readable. This change
alters a number of those.

commit 2b28a87cd9b39e145e6bfcd0b04ed63221dad271
Author: Kenneth Knowles 
Date:   2016-07-29T01:44:39Z

Make DoFn use instance-based TypeDescriptor

commit 5a95226719831e19f86703ac9838bbb5ec2c2362
Author: Kenneth Knowles 
Date:   2016-07-29T01:47:04Z

Use input type in coder inference for MapElements and FlatMapElements

Previously, the input TypeDescriptor was unknown, so we would fail
to infer a coder for things like MapElements.of(SimpleFunction)
even if the input PCollection provided a coder for T.

Now, the input type is plumbed appropriately and the coder is inferred.

This required internal changes to explicitly support good display data.
While doing this, I just added display data to SimpleFunction by analogy
with DoFn.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398675#comment-15398675
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/758

[BEAM-498] Rename DoFn to OldDoFn, DoFnWithContext to DoFn, and port some 
examples

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers. Since there was general approval on the mailing list and 
these changes are mostly trivial, I'm mostly just asking Ben for a double-check 
since he has the most technical depth on this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DoFnWithContext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #758


commit a29447cd338825ac056dfe04d2eeadd8fb4a3171
Author: Kenneth Knowles 
Date:   2016-07-22T20:00:10Z

Rename DoFn to OldDoFn

commit 20d6217cf2db355febc637ebe14e113fbef0ce05
Author: Kenneth Knowles 
Date:   2016-07-26T04:27:02Z

Rename NoOpDoFn to NoOpOldDoFn

commit 9026589010efcdb29543d551a6e8ed9117ea20a7
Author: Kenneth Knowles 
Date:   2016-07-22T21:10:01Z

Rename DoFnWithContext to DoFn

commit 93fab04cc9dc6c93e3080ba5ebbc4aba18851758
Author: Kenneth Knowles 
Date:   2016-07-22T21:28:28Z

Port WordCount example from OldDoFn to DoFn

commit a0d40c6eece48231df2247faec8008bb3abc5bb7
Author: Kenneth Knowles 
Date:   2016-07-22T21:28:42Z

Port MinimalWordCount example from OldDoFn to DoFn

commit 0d4a470af28177af9e8956ddc74f05f52b0fb05b
Author: Kenneth Knowles 
Date:   2016-07-22T21:29:01Z

Port WindowedWordCount example from OldDoFn to DoFn

commit cabea9d4241cfd44996f1d58e8032c60f6413dae
Author: Kenneth Knowles 
Date:   2016-07-22T21:29:18Z

Port DebuggingWordCount example from OldDoFn to DoFn

commit 8f0ce4f1edbdd25fe4c408cb1215190806d874ba
Author: Kenneth Knowles 
Date:   2016-07-22T21:29:37Z

Port AutoComplete example from OldDoFn to DoFn

commit 343d3bc75d04cac554d858c238bb21f0df982c9a
Author: Kenneth Knowles 
Date:   2016-07-22T21:29:51Z

Port microbenchmarks to new vocabulary




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #754: [BEAM-156] Use Quiescence to Drive the Dir...

2016-07-28 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/754

[BEAM-156] Use Quiescence to Drive the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
The Executor should add work whenever it becomes Quiescent.

Track the amount of outstanding work in the executor, and
modify the state appropriately whenever work is scheduled or completes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam quiescence_driver

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #754


commit b4afedad05291585205c81e813c7bbb7f0e5c167
Author: Thomas Groh 
Date:   2016-07-22T20:47:19Z

Apply ExecutorUpdates in two Phases

This removes the need for an explicit break by ensuring that work added
by the monitor will not complete and add more work for the monitor to
complete.

commit 3c4a0d39b6655228cfb0d6870e69f229a85537ac
Author: Thomas Groh 
Date:   2016-07-23T01:01:41Z

Add handleEmpty to CompletionCallback

This is invoked when a Transform Executor has no work to do. Usually
this is due to reinvocation of a Source.

commit 292276cfd4e56bfb9a0278e2717d5eb49e304e3e
Author: Thomas Groh 
Date:   2016-07-26T16:53:22Z

Add ProducedOutput method to TransformResult

This can communicate that a PTransform that produced no outputs still
should cause pending work to be evaluated. PCollectionViews modifiy the
state of the evaluator and can cause formerly blocked PTransforms to be
able to progress.

commit 0397f9f67ee1af8866b72019f0b5cf97c4e6b62a
Author: Thomas Groh 
Date:   2016-07-22T20:47:43Z

Use the State of the Executor to drive progress

Add the concept of Quiescence to ExecutorServiceParallelExecutor.

If the executor is Quiescent, it should interrogate root nodes for
additional work. If not, runs of the monitor should update the state as
appropriate.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-156) Implement Quiescence Signalling in the InProcessPipelineRunner

2016-07-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398500#comment-15398500
 ] 

ASF GitHub Bot commented on BEAM-156:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/754

[BEAM-156] Use Quiescence to Drive the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
The Executor should add work whenever it becomes Quiescent.

Track the amount of outstanding work in the executor, and
modify the state appropriately whenever work is scheduled or completes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam quiescence_driver

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #754


commit b4afedad05291585205c81e813c7bbb7f0e5c167
Author: Thomas Groh 
Date:   2016-07-22T20:47:19Z

Apply ExecutorUpdates in two Phases

This removes the need for an explicit break by ensuring that work added
by the monitor will not complete and add more work for the monitor to
complete.

commit 3c4a0d39b6655228cfb0d6870e69f229a85537ac
Author: Thomas Groh 
Date:   2016-07-23T01:01:41Z

Add handleEmpty to CompletionCallback

This is invoked when a Transform Executor has no work to do. Usually
this is due to reinvocation of a Source.

commit 292276cfd4e56bfb9a0278e2717d5eb49e304e3e
Author: Thomas Groh 
Date:   2016-07-26T16:53:22Z

Add ProducedOutput method to TransformResult

This can communicate that a PTransform that produced no outputs still
should cause pending work to be evaluated. PCollectionViews modifiy the
state of the evaluator and can cause formerly blocked PTransforms to be
able to progress.

commit 0397f9f67ee1af8866b72019f0b5cf97c4e6b62a
Author: Thomas Groh 
Date:   2016-07-22T20:47:43Z

Use the State of the Executor to drive progress

Add the concept of Quiescence to ExecutorServiceParallelExecutor.

If the executor is Quiescent, it should interrogate root nodes for
additional work. If not, runs of the monitor should update the state as
appropriate.




> Implement Quiescence Signalling in the InProcessPipelineRunner
> --
>
> Key: BEAM-156
> URL: https://issues.apache.org/jira/browse/BEAM-156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> A pipeline is quiescent when the following two properties hold:
>   There are no triggers that can fire, given the current processing time and 
> watermark
>   All pending elements cannot make progress until a side input produces 
> additional output
> This is approximately equivalent to: If no more input is received, the 
> pipeline will not perform any additional processing absent advances in 
> processing time or event time
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam-site git commit: This closes #28

2016-07-28 Thread frances

This closes #28


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a64faf28
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a64faf28
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a64faf28

Branch: refs/heads/asf-site
Commit: a64faf2857a72e652c5e283945f08656826d35ec
Parents: ea05f97 8d9a8eb
Author: Frances Perry 
Authored: Thu Jul 28 17:27:15 2016 -0700
Committer: Frances Perry 
Committed: Thu Jul 28 17:27:15 2016 -0700

--
 .travis.yml | 9 +
 1 file changed, 9 insertions(+)
--

[1/2] incubator-beam-site git commit: Adding Travis CI YAML config

2016-07-28 Thread frances

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site ea05f978a -> a64faf285


Adding Travis CI YAML config


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/8d9a8eb0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/8d9a8eb0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/8d9a8eb0

Branch: refs/heads/asf-site
Commit: 8d9a8eb043931ee0d91c43803080c8d26c5ef123
Parents: ea05f97
Author: James Malone 
Authored: Mon Jul 25 09:51:14 2016 -0700
Committer: Frances Perry 
Committed: Thu Jul 28 17:25:29 2016 -0700

--
 .travis.yml | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8d9a8eb0/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 000..c305557
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,9 @@
+language: ruby
+rvm:
+  - 2.1
+script:
+  - bundle exec jekyll build
+  - bundle exec htmlproofer ./content --allow-hash-href --check-html 
--file-ignore /javadoc/
+env:
+  global:
+- NOKOGIRI_USE_SYSTEM_LIBRARIES=true # speeds up installation of 
html-proofer
\ No newline at end of file

[GitHub] incubator-beam-site pull request #28: Adding Travis CI YAML config

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/28


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Implement Aggregators directly for direct-java

2016-07-28 Thread bchambers

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9c447510a -> 1df6f5f97


Implement Aggregators directly for direct-java

Previously, we relied on conversion to Counter rather than just
running the specified CombineFn. This aligns the direct runner
more closely with the model.

This PR also parameterizes DoFnRunner on an AggregatorFactory to
implement aggregators, allowing each runner to provide the appropriate
implementation.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/15eb67bb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/15eb67bb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/15eb67bb

Branch: refs/heads/master
Commit: 15eb67bb06f7cf0f225249810df8b521649f4f23
Parents: 9c44751
Author: bchambers 
Authored: Tue Jul 26 10:33:40 2016 -0700
Committer: bchambers 
Committed: Thu Jul 28 17:20:44 2016 -0700

--
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  28 +--
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  92 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |   7 +-
 .../runners/direct/AggregatorContainer.java | 183 +++
 .../beam/runners/direct/DirectRunner.java   |  14 +-
 .../beam/runners/direct/EvaluationContext.java  |  32 ++--
 .../beam/runners/direct/ParDoEvaluator.java |  16 +-
 .../runners/direct/StepTransformResult.java |  15 +-
 .../beam/runners/direct/TransformResult.java|   9 +-
 .../runners/direct/AggregatorContainerTest.java | 134 ++
 .../beam/runners/direct/DirectRunnerTest.java   |   4 -
 .../runners/direct/EvaluationContextTest.java   |  35 ++--
 .../beam/runners/direct/ParDoEvaluatorTest.java |  11 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  |  31 ++--
 .../direct/ParDoSingleEvaluatorFactoryTest.java |  31 ++--
 .../apache/beam/sdk/transforms/Aggregator.java  |  24 ++-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   7 +-
 .../apache/beam/sdk/util/CounterAggregator.java |  35 +++-
 .../apache/beam/sdk/transforms/DoFnTest.java|  35 
 19 files changed, 592 insertions(+), 151 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/15eb67bb/runners/core-java/src/main/java/org/apache/beam/sdk/util/DoFnRunnerBase.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/sdk/util/DoFnRunnerBase.java 
b/runners/core-java/src/main/java/org/apache/beam/sdk/util/DoFnRunnerBase.java
index e267a31..2696020 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/sdk/util/DoFnRunnerBase.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/sdk/util/DoFnRunnerBase.java
@@ -23,6 +23,7 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.IterableCoder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.beam.sdk.transforms.Aggregator.AggregatorFactory;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.RequiresWindowAccess;
@@ -33,19 +34,15 @@ import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.DoFnRunners.OutputManager;
 import org.apache.beam.sdk.util.ExecutionContext.StepContext;
-import org.apache.beam.sdk.util.common.CounterSet;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
-
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
-
 import org.joda.time.Instant;
 import org.joda.time.format.PeriodFormat;
-
 import java.io.IOException;
 import java.util.Collection;
 import java.util.Collections;
@@ -75,7 +72,7 @@ public abstract class DoFnRunnerBase 
implements DoFnRunner mainOutputTag,
   List sideOutputTags,
   StepContext stepContext,
-  CounterSet.AddCounterMutator addCounterMutator,
+  AggregatorFactory aggregatorFactory,
   WindowingStrategy windowingStrategy) {
 this.fn = fn;
 this.context = new DoFnContext<>(
@@ -86,13 +83,13 @@ public abstract class DoFnRunnerBase 
implements DoFnRunner 
implements DoFnRunner mainOutputTag;
 final StepContext stepContext;
-final CounterSet.AddCounterMutator addCounterMutator;
+final AggregatorFactory aggregatorFactory;
 final WindowFn windowFn;
 
 /**
@@ -196,7 +193,7 @@ public abstract class DoFnRunnerBase 
implements DoFnRunner

[2/2] incubator-beam git commit: Implement aggregators directly in the direct runner

2016-07-28 Thread bchambers

Implement aggregators directly in the direct runner

This closes #735


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1df6f5f9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1df6f5f9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1df6f5f9

Branch: refs/heads/master
Commit: 1df6f5f976108aa584abbb8a9412578e1f6a621b
Parents: 9c44751 15eb67b
Author: bchambers 
Authored: Thu Jul 28 17:22:03 2016 -0700
Committer: bchambers 
Committed: Thu Jul 28 17:22:03 2016 -0700

--
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  28 +--
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  92 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |   7 +-
 .../runners/direct/AggregatorContainer.java | 183 +++
 .../beam/runners/direct/DirectRunner.java   |  14 +-
 .../beam/runners/direct/EvaluationContext.java  |  32 ++--
 .../beam/runners/direct/ParDoEvaluator.java |  16 +-
 .../runners/direct/StepTransformResult.java |  15 +-
 .../beam/runners/direct/TransformResult.java|   9 +-
 .../runners/direct/AggregatorContainerTest.java | 134 ++
 .../beam/runners/direct/DirectRunnerTest.java   |   4 -
 .../runners/direct/EvaluationContextTest.java   |  35 ++--
 .../beam/runners/direct/ParDoEvaluatorTest.java |  11 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  |  31 ++--
 .../direct/ParDoSingleEvaluatorFactoryTest.java |  31 ++--
 .../apache/beam/sdk/transforms/Aggregator.java  |  24 ++-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   7 +-
 .../apache/beam/sdk/util/CounterAggregator.java |  35 +++-
 .../apache/beam/sdk/transforms/DoFnTest.java|  35 
 19 files changed, 592 insertions(+), 151 deletions(-)
--

[GitHub] incubator-beam pull request #735: Implement Aggregators directly in the Dire...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/735


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-498) Make DoFnWithContext the new DoFn

2016-07-28 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-498:


 Summary: Make DoFnWithContext the new DoFn
 Key: BEAM-498
 URL: https://issues.apache.org/jira/browse/BEAM-498
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-487) Add disclaimer to GitHub README.md

2016-07-28 Thread Daniel Halperin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-487.

Assignee: Daniel Halperin  (was: Jean-Baptiste Onofré)

> Add disclaimer to GitHub README.md
> --
>
> Key: BEAM-487
> URL: https://issues.apache.org/jira/browse/BEAM-487
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: 0.2.0-incubating
>
>
> http://mail-archives.apache.org/mod_mbox/incubator-general/201606.mbox/%3c47b44d85-be01-42f9-96c6-43ff23e31...@apache.org%3E
> > 1. The DISCLAIMER file is sufficient for the purposes of a source release. 
> > But for the github
> audience, reaching https://github.com/apache/incubator-beam 
> ,
> there is no disclaimer. I think there should be a disclaimer on README.md, 
> and at least the
> first reference to beam should read “Apache Beam (incubating)”.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-488) Remove KEYS file

2016-07-28 Thread Daniel Halperin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-488.

Assignee: Daniel Halperin  (was: Jean-Baptiste Onofré)

> Remove KEYS file
> 
>
> Key: BEAM-488
> URL: https://issues.apache.org/jira/browse/BEAM-488
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: 0.2.0-incubating
>
>
> http://mail-archives.apache.org/mod_mbox/incubator-general/201606.mbox/%3CCAAS6=7hVLcw6060Un7sXxk+WLLh08DFOSWktC0Aam4F=dye...@mail.gmail.com%3E
> > Bundling PGP keys inside a package is worse than worthless -- an attacker 
> > can
> just bundle spoofed keys with a bogus distro!  Keys need to be made available
> from a highly reliable, separate server: Download the main package from a
> mirror, get PGP keys from apache.org, pgp.mit.edu, etc. and verify.
> > 
> > The KEYS file within the Beam source tree should be deleted.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-486) Cleanup NOTICE file

2016-07-28 Thread Daniel Halperin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-486.

Assignee: Daniel Halperin  (was: Jean-Baptiste Onofré)

> Cleanup NOTICE file
> ---
>
> Key: BEAM-486
> URL: https://issues.apache.org/jira/browse/BEAM-486
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: 0.2.0-incubating
>
>
> http://mail-archives.apache.org/mod_mbox/incubator-general/201606.mbox/%3ca5f50a0f-f1e1-4391-8188-391187b9e...@classsoftware.com%3E
> - NOTICE file contain unneeded text (i.e mentions  Apache v2.0 licence). 
> There no need to
> generally mention Apache 2.0 licences in NOTICE [2]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #752: Beam Python GCSIO write performance improv...

2016-07-28 Thread mdvorsky

GitHub user mdvorsky opened a pull request:

https://github.com/apache/incubator-beam/pull/752

Beam Python GCSIO write performance improvements.

Increased the GCS buffer size from 1MB to 8MB and introduced a 128kB buffer 
for the pipe. 

Increasing the GCS buffer even more could improve the performance, but for 
the price of spending a lot of memory on buffers (which can be a problem if 
many GCS files are being written to concurrently).

Increasing the pipe buffer further appears to have no benefit.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mdvorsky/incubator-beam fast_gcsio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #752


commit a1f1fa06ee8683273182548e7eb2d6612040d2bf
Author: Marian Dvorsky 
Date:   2016-07-28T20:02:15Z

Increased the GCS buffer size from 1MB to 8MB and introduced a 128kB buffer 
for the pipe.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #726: Make TextFileReader observable

2016-07-28 Thread charlesccychen

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/726


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #751: Fix hashing and comparison for compression...

2016-07-28 Thread silviulica

GitHub user silviulica opened a pull request:

https://github.com/apache/incubator-beam/pull/751

Fix hashing and comparison for compression types

R: @robertwb Please take a look Robert. Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/silviulica/incubator-beam compression_types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #751


commit d57028b9e9ff4917f868cc12a14c7842737cf3c6
Author: Silviu Calinoiu 
Date:   2016-07-28T16:05:50Z

Fix hashing and comparison for compression types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #747: Remove unneccessary method in ReadEvaluato...

2016-07-28 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/747


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

66 matches

Mail list logo