[jira] [Commented] (BEAM-485) Can't set Flink runner in code

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438508#comment-15438508
 ] 

Aljoscha Krettek commented on BEAM-485:
---

Hi [~ecesena], could you check whether this bug is still valid in the current 
master version?

> Can't set Flink runner in code
> --
>
> Key: BEAM-485
> URL: https://issues.apache.org/jira/browse/BEAM-485
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Emanuele Cesena
>
> Calling:
> options.setRunner(FlinkRunner.class);
> doesn't seem to properly set the runner.
> Running --runner=FlinkRunner from the command line works.
> Both approaches were working on 0.1.0, but options.setRunner doesn't seem to 
> work on master anymore.
> I noticed there are tests that only cover the command line case:
> https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/FlinkRunnerRegistrarTest.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-253) Unify Flink Operator Wrappers

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek resolved BEAM-253.
---
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

Implemented here: 
https://github.com/apache/incubator-beam/commit/1de76b7a5169a46ef9f14406e5a6e1284832f7f9

> Unify Flink Operator Wrappers
> -
>
> Key: BEAM-253
> URL: https://issues.apache.org/jira/browse/BEAM-253
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> Right now, we have {{FlinkAbstractParDoWrapper}} with subclasses 
> {{FlinkParDoBoundWrapper}} and {{FlinkParDoBoundMultiWrapper}} as well as 
> {{FlinkGroupAlsoByWindowWrapper}}. They do essentially the same thing, but 
> slightly differently. The first three are implemented as a 
> {{FlatMapFunction}} while the latter is implemented as a {{StreamOperator}} 
> (which is more low-level and gives access to state and timers and such).
> We should unify this into one wrapper. (With possibly a more concise name...)
> In the process of this we should also make sure that we always use a 
> {{DoFnRunner}} via {{DoFnRunners.createDefault}}. This will help reduce bugs 
> such as [BEAM-241].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-315) Flink Runner compares keys unencoded which may produce incorrect results

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek resolved BEAM-315.
---
Resolution: Fixed

> Flink Runner compares keys unencoded which may produce incorrect results
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Pawel Szczur
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
> Attachments: CoGroupPipelineStringKey.java, execution.log, 
> execution_split.log, execution_split_sorted.log
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-102) Side Inputs for Streaming

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek resolved BEAM-102.
---
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

Implemented 
here:https://github.com/apache/incubator-beam/commit/dfbdc6c2bbef5e749bfc1800f97d21377f0c713d

> Side Inputs for Streaming
> -
>
> Key: BEAM-102
> URL: https://issues.apache.org/jira/browse/BEAM-102
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> The Flink Runner supports side inputs for batch mode but its missing support 
> for streaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #142

[Apache Jenkins Server]

See 


--
[...truncated 9298 lines...]
[INFO] Including io.grpc:grpc-protobuf:jar:0.14.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.0.0-beta-2 in the 
shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.3 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:0.14.1 in the shaded jar.
[INFO] Including io.grpc:grpc-okhttp:jar:0.14.1 in the shaded jar.
[INFO] Including com.squareup.okio:okio:jar:1.6.0 in the shaded jar.
[INFO] Including com.squareup.okhttp:okhttp:jar:2.5.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:0.14.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-nano:jar:0.14.1 in the shaded jar.
[INFO] Including com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
in the shaded jar.
[INFO] Including com.google.auth:google-auth-library-oauth2-http:jar:0.4.0 in 
the shaded jar.
[INFO] Including com.google.auth:google-auth-library-credentials:jar:0.4.0 in 
the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.1.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.1.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.1.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.1.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.1.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.1.Final in the shaded jar.
[INFO] Including com.google.api.grpc:grpc-pubsub-v1:jar:0.0.2 in the shaded jar.
[INFO] Including com.google.api.grpc:grpc-core-proto:jar:0.0.3 in the shaded 
jar.
[INFO] Including com.google.api-client:google-api-client:jar:1.22.0 in the 
shaded jar.
[INFO] Including 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 in the shaded 
jar.
[INFO] Including com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
in the shaded jar.
[INFO] Including 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 in the shaded 
jar.
[INFO] Including com.google.http-client:google-http-client:jar:1.22.0 in the 
shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.0.1 in the shaded 
jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.0.1 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.3 in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-jackson:jar:1.22.0 
in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-jackson2:jar:1.22.0 
in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-protobuf:jar:1.22.0 
in the shaded jar.
[INFO] Including com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
in the shaded jar.
[INFO] Including com.google.oauth-client:google-oauth-client:jar:1.22.0 in the 
shaded jar.
[INFO] Including com.google.cloud.bigdataoss:gcsio:jar:1.4.5 in the shaded jar.
[INFO] Including com.google.api-client:google-api-client-java6:jar:1.22.0 in 
the shaded jar.
[INFO] Including com.google.api-client:google-api-client-jackson2:jar:1.22.0 in 
the shaded jar.
[INFO] Including com.google.cloud.bigdataoss:util:jar:1.4.5 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0-beta-1 in the 
shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.7.2 in the 
shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 in the 
shaded jar.
[INFO] Including net.bytebuddy:byte-buddy:jar:1.4.3 in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.8.1 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded 
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the 
shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.1 in the shaded jar.
[INFO] Including org.apache.commons:commons-compress:jar:1.9 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2 in 
the shaded jar.
[INFO] Including com.google.code.findbugs:annotations:jar:3.0.1 in the shaded 
jar.
[INFO] Including net.jcip:jcip-annotations:jar:1.0 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.1 in the shaded jar.
[INFO] Excluding com.google.guava:guava:jar:19.0 from the shaded jar.
[INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.14 in the shaded jar.
[WARNING] jsr305-3.0.1.jar, annotations-3.0.1.jar define 35 overlapping 
classes: 
[WARNING]   - javax.annotation.RegEx
[WARNING]   - javax.annotation.concurrent.Immutable
[WARNING]   - javax.annotation.meta.TypeQualifierDefault
[WARNING]   - 

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438364#comment-15438364
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/750


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #750: [BEAM-79] Merge branch 'master' into gearp...

[manuzhang]

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/750


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #889: [BEAM-589] Fixing IO.Read transformation

[gauravgopi123]

GitHub user gauravgopi123 opened a pull request:

https://github.com/apache/incubator-beam/pull/889

[BEAM-589] Fixing IO.Read transformation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gauravgopi123/incubator-beam BEAM-589

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/889.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #889


commit 2438d2b9e1fef6f5e8742fa65fe70d593f2f6145
Author: gaurav gupta 
Date:   2016-08-25T21:00:06Z

[BEAM-589] Fixing IO.Read transformation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-589) Making IO.Read transformations to extend PTransform<PBegin, PCollection>

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438174#comment-15438174
 ] 

ASF GitHub Bot commented on BEAM-589:
-

GitHub user gauravgopi123 opened a pull request:

https://github.com/apache/incubator-beam/pull/889

[BEAM-589] Fixing IO.Read transformation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gauravgopi123/incubator-beam BEAM-589

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/889.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #889


commit 2438d2b9e1fef6f5e8742fa65fe70d593f2f6145
Author: gaurav gupta 
Date:   2016-08-25T21:00:06Z

[BEAM-589] Fixing IO.Read transformation




> Making IO.Read transformations to extend PTransform
> ---
>
> Key: BEAM-589
> URL: https://issues.apache.org/jira/browse/BEAM-589
> Project: Beam
>  Issue Type: Bug
>Reporter: gaurav gupta
>
> Currently TextIO.Read.Bound, AvroIO.Read.Bound, PubsubIO.Read.Bound, 
> BIgQueryIO.Read.Bound are all extending from PTransform PCollection>  instead they should be extending from  PTransform PCollection>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #141

[Apache Jenkins Server]

See 


--
[...truncated 10846 lines...]
at 
org.apache.beam.runners.gearpump.GearpumpPipelineTranslator.translate(GearpumpPipelineTranslator.java:83)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.run(GearpumpPipelineRunner.java:111)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.run(TestGearpumpRunner.java:54)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.run(TestGearpumpRunner.java:34)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:185)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:117)
at 
org.apache.beam.sdk.transforms.ApproximateUniqueTest.testApproximateUniqueWithSmallInput(ApproximateUniqueTest.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:387)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Running org.apache.beam.sdk.transforms.windowing.WindowingTest
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.323 sec <<< 
FAILURE! - in org.apache.beam.sdk.transforms.windowing.WindowingTest
testMergingWindowing(org.apache.beam.sdk.transforms.windowing.WindowingTest)  
Time elapsed: 0.958 sec  <<< ERROR!
java.lang.UnsupportedOperationException: merging window is not supported in 
Gearpump pipeline
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner$AssignWindowsAndSetStrategy.apply(GearpumpPipelineRunner.java:167)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner$AssignWindowsAndSetStrategy.apply(GearpumpPipelineRunner.java:149)
at 
org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.apply(GearpumpPipelineRunner.java:81)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.apply(TestGearpumpRunner.java:62)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:403)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:326)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:167)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest$WindowedCount.apply(WindowingTest.java:76)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest$WindowedCount.apply(WindowingTest.java:58)
at 
org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at 
org.apache.beam.runners.gearpump.GearpumpPipelineRunner.apply(GearpumpPipelineRunner.java:93)
at 
org.apache.beam.runners.gearpump.TestGearpumpRunner.apply(TestGearpumpRunner.java:62)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:403)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:310)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at 
org.apache.beam.sdk.transforms.windowing.WindowingTest.testMergingWindowing(WindowingTest.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 

[GitHub] incubator-beam pull request #872: checkstyle: prohibit API client repackaged...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/872


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #872

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9c9f4c9c9 -> bfd810f41


Closes #872


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bfd810f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bfd810f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bfd810f4

Branch: refs/heads/master
Commit: bfd810f410140a2fc5e6099755e58ae444a4e023
Parents: 9c9f4c9 89cf461
Author: Dan Halperin 
Authored: Thu Aug 25 16:11:21 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 16:11:21 2016 -0700

--
 runners/spark/pom.xml |  4 
 .../streaming/StreamingTransformTranslator.java   |  6 +++---
 .../build-tools/src/main/resources/beam/checkstyle.xml| 10 +-
 .../apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java   |  2 +-
 sdks/java/io/hdfs/pom.xml |  5 -
 .../java/org/apache/beam/sdk/io/hdfs/HDFSFileSink.java|  2 +-
 6 files changed, 14 insertions(+), 15 deletions(-)
--

[2/2] incubator-beam git commit: checkstyle: prohibit API client repackaged Guava

[dhalperi]

checkstyle: prohibit API client repackaged Guava

Apparently the IllegalImport check only blocks packages,
so we had to move to Regexp to get individual classes.

As a bonus, this enforcement let us remove two bogus dependencies.
Smaller JARs for the win!


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/89cf4613
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/89cf4613
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/89cf4613

Branch: refs/heads/master
Commit: 89cf4613465647e2711983674879afd5f67c519d
Parents: 9c9f4c9
Author: Dan Halperin 
Authored: Tue Aug 23 18:54:23 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 16:11:21 2016 -0700

--
 runners/spark/pom.xml |  4 
 .../streaming/StreamingTransformTranslator.java   |  6 +++---
 .../build-tools/src/main/resources/beam/checkstyle.xml| 10 +-
 .../apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java   |  2 +-
 sdks/java/io/hdfs/pom.xml |  5 -
 .../java/org/apache/beam/sdk/io/hdfs/HDFSFileSink.java|  2 +-
 6 files changed, 14 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89cf4613/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index a6311b5..a5e99a0 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -155,10 +155,6 @@
   true
 
 
-  com.google.http-client
-  google-http-client
-
-
   com.fasterxml.jackson.core
   jackson-core
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89cf4613/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
index 43160f5..5f35ebb 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
@@ -17,9 +17,9 @@
  */
 package org.apache.beam.runners.spark.translation.streaming;
 
-import com.google.api.client.util.Lists;
-import com.google.api.client.util.Maps;
-import com.google.api.client.util.Sets;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
 import com.google.common.reflect.TypeToken;
 import java.lang.reflect.ParameterizedType;
 import java.lang.reflect.Type;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89cf4613/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml 
b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index 47ddc5b..c7d9b2c 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -100,7 +100,15 @@ page at http://checkstyle.sourceforge.net/config.html -->
 
 
 
-  
+  
+
+
+
+
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89cf4613/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
index a6d7e2f..7a7575b 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
@@ -33,7 +33,6 @@ import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.when;
 
 import com.google.api.client.util.Data;
-import com.google.api.client.util.Strings;
 import com.google.api.services.bigquery.model.ErrorProto;
 import com.google.api.services.bigquery.model.Job;
 import com.google.api.services.bigquery.model.JobConfigurationExtract;
@@ -50,6 +49,7 @@ import 
com.google.api.services.bigquery.model.TableFieldSchema;
 import 

[GitHub] incubator-beam pull request #886: Make WriteTest more resilient to Randomnes...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/886


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Make WriteTest more resilient to Randomness

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master edcb5eff3 -> 9c9f4c9c9


Make WriteTest more resilient to Randomness

In the worst case scenario for random key assignment in
Write.ApplyShardingKey, the chance of the number of records per output
shard was too high. This makes the test significantly less likely to
flake.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/81352b42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/81352b42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/81352b42

Branch: refs/heads/master
Commit: 81352b4264999a4a26da8be0bb93bf3a218354d4
Parents: edcb5ef
Author: Thomas Groh 
Authored: Thu Aug 25 14:58:56 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 15:40:07 2016 -0700

--
 .../core/src/test/java/org/apache/beam/sdk/io/WriteTest.java  | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/81352b42/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
--
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
index 2865188..997566a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
@@ -192,11 +192,12 @@ public class WriteTest {
   inputs.add(String.format("elt%04d", i));
 }
 
+int numShards = 10;
 runShardedWrite(
 inputs,
 new WindowAndReshuffle<>(
 Window.into(Sessions.withGapDuration(Duration.millis(1,
-Optional.of(10));
+Optional.of(numShards));
 
 // Check that both the min and max number of results per shard are close 
to the expected.
 int min = Integer.MAX_VALUE;
@@ -205,7 +206,9 @@ public class WriteTest {
   min = Math.min(min, i);
   max = Math.max(max, i);
 }
-assertThat((double) min, Matchers.greaterThanOrEqualTo(max * 0.9));
+double expected = numElements / (double) numShards;
+assertThat((double) min, Matchers.greaterThanOrEqualTo(expected * 0.6));
+assertThat((double) max, Matchers.lessThanOrEqualTo(expected * 1.4));
   }
 
   /**

[GitHub] incubator-beam pull request #888: [Draft] Cleaner backoff utilities in Beam

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/888

[Draft] Cleaner backoff utilities in Beam

We have 3 different backoff classes, which don't really have that much 
different functionality. Add a single, flexible backoff implementation that can 
be used to replace all three classes. Additionally, this new backoff actually 
supports more functionality than any of the other three did -- you can limit 
retries, cap the exponential growth of an individual backoff, and cap the 
cumulative time spent in backoff; prior implementations did not allow all 3.

This also makes the parameters self-obvious (`Duration`, not 
number-that-is-also-millis) where appropriate.

This initial PR should have no functional changes.

TODO:
- [ ] javadoc on FlexibleBackof
- [ ] is there a better name than "FlexibleBackoff"

?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam cleaner-backoff

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/888.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #888


commit c98304a0ca574e93303f07a8a0414269f7656be9
Author: Dan Halperin 
Date:   2016-08-25T05:35:26Z

FlexibleBackoff: a replacement for a variety of custom backoff 
implementations

commit 3a35771deda3a022c84591b23b1c5b24958ec121
Author: Dan Halperin 
Date:   2016-08-25T22:27:52Z

Replace other custom BackOff implementations with FlexibleBackoff




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-194) Create a walkthrough of Beam examples in mobile gaming domain

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437795#comment-15437795
 ] 

Pei He commented on BEAM-194:
-

And,
https://cloud.google.com/dataflow/model/triggers
https://cloud.google.com/dataflow/model/windowing#Advanced

> Create a walkthrough of Beam examples in mobile gaming domain
> -
>
> Key: BEAM-194
> URL: https://issues.apache.org/jira/browse/BEAM-194
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Devin Donnelly
>
> The Beam SDKs provide a series of example pipelines in the mobile gaming 
> domain. The Dataflow documentation contains an detailed walkthrough of these 
> examples, explaining the use case, pipeline design, and some of the code.
> Port these examples to the Beam website for Beam users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437790#comment-15437790
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/887

[BEAM-362] Put classes in runners-core package into runners.core namespace

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @dhalperi 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam runners-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/887.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #887


commit e46886e7adbacdb3edd65c1b8f3aa5b488660719
Author: Kenneth Knowles 
Date:   2016-08-25T21:57:26Z

Put classes in runners-core package into runners.core namespace




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #887: [BEAM-362] Put classes in runners-core pac...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/887

[BEAM-362] Put classes in runners-core package into runners.core namespace

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @dhalperi 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam runners-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/887.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #887


commit e46886e7adbacdb3edd65c1b8f3aa5b488660719
Author: Kenneth Knowles 
Date:   2016-08-25T21:57:26Z

Put classes in runners-core package into runners.core namespace




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #886: Make WriteTest more resilient to Randomnes...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/886

Make WriteTest more resilient to Randomness

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

In the worst case scenario for random key assignment in
Write.ApplyShardingKey, the chance of the number of records per output
shard was too high. This makes the test significantly less likely to
flake.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam balance_resilience

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/886.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #886


commit ebafc8154ce8d2ad981e060042a74858e1a419ce
Author: Thomas Groh 
Date:   2016-08-25T21:58:56Z

Make WriteTest more resilient to Randomness

In the worst case scenario for random key assignment in
Write.ApplyShardingKey, the chance of the number of records per output
shard was too high. This makes the test significantly less likely to
flake.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #844

[kenn]

This closes #844


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/edcb5eff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/edcb5eff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/edcb5eff

Branch: refs/heads/master
Commit: edcb5eff3f6e7182954631ff0e21303493858958
Parents: a69a0ea 4e6230c
Author: Kenneth Knowles 
Authored: Thu Aug 25 14:50:17 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Aug 25 14:50:17 2016 -0700

--
 .../examples/common/PubsubFileInjector.java |   2 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  16 +-
 .../beam/sdk/util/GroupAlsoByWindowsDoFn.java   |   2 +-
 .../apache/beam/sdk/util/ReduceFnRunner.java|   5 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |   4 +-
 .../ImmutabilityCheckingBundleFactory.java  |   4 +-
 .../direct/TransformEvaluatorFactory.java   |   3 +-
 .../beam/runners/dataflow/util/DoFnInfo.java|   7 +-
 .../translation/MultiOutputWordCountTest.java   |   2 +-
 .../spark/translation/SerializationTest.java|   4 +-
 .../org/apache/beam/sdk/AggregatorValues.java   |   4 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |  14 +-
 .../apache/beam/sdk/transforms/CombineFns.java  |  18 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|  23 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |  62 ++--
 .../apache/beam/sdk/transforms/GroupByKey.java  |   7 +-
 .../apache/beam/sdk/transforms/PTransform.java  |   2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   | 306 +--
 .../beam/sdk/transforms/SimpleFunction.java |   6 +-
 .../beam/sdk/transforms/windowing/PaneInfo.java |  10 +-
 .../beam/sdk/util/BaseExecutionContext.java |   4 +-
 .../sdk/util/ReifyTimestampAndWindowsDoFn.java  |   4 +-
 .../apache/beam/sdk/util/SerializableUtils.java |   2 +-
 .../beam/sdk/util/SystemDoFnInternal.java   |   7 +-
 .../beam/sdk/util/WindowingInternals.java   |   3 +-
 .../DoFnDelegatingAggregatorTest.java   |   2 +-
 .../beam/sdk/transforms/DoFnTesterTest.java |   3 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |   2 +-
 28 files changed, 263 insertions(+), 265 deletions(-)
--

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437769#comment-15437769
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/844


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-590) Port examples web docs from Dataflow to Beam website.

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He resolved BEAM-590.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> Port examples web docs from Dataflow to Beam website.
> -
>
> Key: BEAM-590
> URL: https://issues.apache.org/jira/browse/BEAM-590
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> I am removing references to dataflow website in examples, such as:
> https://cloud.google.com/dataflow/java-sdk/wordcount-example
> Creating this issue to track web docs that we might want to port to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-590) Port examples web docs from Dataflow to Beam website.

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437741#comment-15437741
 ] 

Pei He commented on BEAM-590:
-

We can merge them under the same jira issue.
Closing this one.

> Port examples web docs from Dataflow to Beam website.
> -
>
> Key: BEAM-590
> URL: https://issues.apache.org/jira/browse/BEAM-590
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> I am removing references to dataflow website in examples, such as:
> https://cloud.google.com/dataflow/java-sdk/wordcount-example
> Creating this issue to track web docs that we might want to port to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-194) Create a walkthrough of Beam examples in mobile gaming domain

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437739#comment-15437739
 ] 

Pei He commented on BEAM-194:
-

We also have the dataflow word count walk through that it is referenced in Beam 
examples code:
https://cloud.google.com/dataflow/java-sdk/wordcount-example

I am removing those links from Beam examples. 

> Create a walkthrough of Beam examples in mobile gaming domain
> -
>
> Key: BEAM-194
> URL: https://issues.apache.org/jira/browse/BEAM-194
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Devin Donnelly
>
> The Beam SDKs provide a series of example pipelines in the mobile gaming 
> domain. The Dataflow documentation contains an detailed walkthrough of these 
> examples, explaining the use case, pipeline design, and some of the code.
> Port these examples to the Beam website for Beam users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #885: Remove the DataflowRunner instructions fro...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/885

Remove the DataflowRunner instructions from examples





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam examples-cleanup-10

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/885.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #885


commit 69c35355d2665b442be7c453ecb3b16bd4cb2f5e
Author: Pei He 
Date:   2016-08-25T21:20:30Z

Remove the DataflowRunner instructions from examples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-591) Better handling watermark in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-591:
--
Description: 
Right now default watermark in KafkaIO is same as timestamp of the record. The 
main problem with this is that watermark does not change if there n't any new 
records on the topic. This can hold up many open windows. 

The record timestamp by default is set to processing time (i.e. when the runner 
reads a record from Kafka reader).

A user can provide functions to calculate watermark and record timestamps. 
There are a few concerns with current design:

* What should happen when a kafka topic is idle:
  ** in default case, I think watermark should advance to current time.
  ** What should happen when user has provided a function to calculate record 
timestamp? 
   *** Should the watermark stay same as record timestamp?
   *** same when user has provided own watermark function? 
* Are the current semantics of user provided watermark function correct?
  ** it is run once for each record read.
  ** Should it instead be run inside {{getWatermark()}} called by the runner 
(we could still provide the last user record, and its timestamp).
 


  was:

Right now default watermark in KafkaIO is same as timestamp of the record. The 
main problem with this is that watermark does not change if there n't any new 
records on the topic. This can hold up many open windows. 

The record timestamp by default is set to processing time (i.e. when the runner 
reads a record from Kafka reader).

A user provide functions to calculate watermark and record timestamps. There 
are a few concerns:

* What should happen when a kafka topic is idle:
  ** in default case, I think watermark should advance to current time.
  ** What should happen when user has provided a function to calculate record 
timestamp? 
   *** Should the watermark stay same as record timestamp?
   *** same when user has provided own watermark function? 
* Are the current semantics of user provided watermark function correct?
  ** it is run once for each record read.
  ** Should it instead be run inside {{getWatermark()}} called by the runner 
(we could still provide the last user record, and its timestamp).
 



> Better handling watermark in KafkaIO
> 
>
> Key: BEAM-591
> URL: https://issues.apache.org/jira/browse/BEAM-591
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>
> Right now default watermark in KafkaIO is same as timestamp of the record. 
> The main problem with this is that watermark does not change if there n't any 
> new records on the topic. This can hold up many open windows. 
> The record timestamp by default is set to processing time (i.e. when the 
> runner reads a record from Kafka reader).
> A user can provide functions to calculate watermark and record timestamps. 
> There are a few concerns with current design:
> * What should happen when a kafka topic is idle:
>   ** in default case, I think watermark should advance to current time.
>   ** What should happen when user has provided a function to calculate record 
> timestamp? 
>*** Should the watermark stay same as record timestamp?
>*** same when user has provided own watermark function? 
> * Are the current semantics of user provided watermark function correct?
>   ** it is run once for each record read.
>   ** Should it instead be run inside {{getWatermark()}} called by the runner 
> (we could still provide the last user record, and its timestamp).
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-591) Better handling watermark in KafkaIO

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-591:
-

 Summary: Better handling watermark in KafkaIO
 Key: BEAM-591
 URL: https://issues.apache.org/jira/browse/BEAM-591
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: Raghu Angadi



Right now default watermark in KafkaIO is same as timestamp of the record. The 
main problem with this is that watermark does not change if there n't any new 
records on the topic. This can hold up many open windows. 

The record timestamp by default is set to processing time (i.e. when the runner 
reads a record from Kafka reader).

A user provide functions to calculate watermark and record timestamps. There 
are a few concerns:

* What should happen when a kafka topic is idle:
  ** in default case, I think watermark should advance to current time.
  ** What should happen when user has provided a function to calculate record 
timestamp? 
   *** Should the watermark stay same as record timestamp?
   *** same when user has provided own watermark function? 
* Are the current semantics of user provided watermark function correct?
  ** it is run once for each record read.
  ** Should it instead be run inside {{getWatermark()}} called by the runner 
(we could still provide the last user record, and its timestamp).
 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-590) Port examples web docs from Dataflow to Beam website.

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437677#comment-15437677
 ] 

Frances Perry commented on BEAM-590:


Is this a dup of BEAM-194?

> Port examples web docs from Dataflow to Beam website.
> -
>
> Key: BEAM-590
> URL: https://issues.apache.org/jira/browse/BEAM-590
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Pei He
>Priority: Minor
>
> I am removing references to dataflow website in examples, such as:
> https://cloud.google.com/dataflow/java-sdk/wordcount-example
> Creating this issue to track web docs that we might want to port to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-589) Making IO.Read transformations to extend PTransform<PBegin, PCollection>

[gaurav gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437663#comment-15437663
 ] 

gaurav gupta commented on BEAM-589:
---

I will take this up. Unfortunately I am not able to assign this to myself

> Making IO.Read transformations to extend PTransform
> ---
>
> Key: BEAM-589
> URL: https://issues.apache.org/jira/browse/BEAM-589
> Project: Beam
>  Issue Type: Bug
>Reporter: gaurav gupta
>
> Currently TextIO.Read.Bound, AvroIO.Read.Bound, PubsubIO.Read.Bound, 
> BIgQueryIO.Read.Bound are all extending from PTransform PCollection>  instead they should be extending from  PTransform PCollection>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-589) Making IO.Read transformations to extend PTransform<PBegin, PCollection>

[gaurav gupta (JIRA)]

gaurav gupta created BEAM-589:
-

 Summary: Making IO.Read transformations to extend 
PTransform
 Key: BEAM-589
 URL: https://issues.apache.org/jira/browse/BEAM-589
 Project: Beam
  Issue Type: Bug
Reporter: gaurav gupta


Currently TextIO.Read.Bound, AvroIO.Read.Bound, PubsubIO.Read.Bound, 
BIgQueryIO.Read.Bound are all extending from PTransform 
 instead they should be extending from  PTransform



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-545) Pipelines and their executions naming changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437634#comment-15437634
 ] 

ASF GitHub Bot commented on BEAM-545:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/719


> Pipelines and their executions naming changes
> -
>
> Key: BEAM-545
> URL: https://issues.apache.org/jira/browse/BEAM-545
> Project: Beam
>  Issue Type: Improvement
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> The purpose of the changes is to clarify the differences between the two, have
> consensus between runners, and unify the implementation.
> Current states:
>  * PipelineOptions.appName defaults to mainClass name
>  * DataflowPipelineOptions.jobName defaults to appName+user+datetime
>  * FlinkPipelineOptions.jobName defaults to appName+user+datetime
> Proposal:
> 1. Replace PipelineOptions.appName with PipelineOptions.pipelineName.
> *  It is the user-visible name for a specific graph.
> *  default to mainClass name.
> *  Use cases: Find all executions of a pipeline
> 2. Add jobName to top level PipelineOptions.
> *  It is the unique name for an execution
> *  defaults to pipelineName + user + datetime + random Integer
> *  Use cases:
> -- Finding all executions by USER_A between TIME_X and TIME_Y
> -- Naming resources created by the execution. for example:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #719: [BEAM-545] Promote JobName to PipelineOpti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/719


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-545] Promote JobName to PipelineOptions

[dhalperi]

[BEAM-545] Promote JobName to PipelineOptions


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b04776d4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b04776d4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b04776d4

Branch: refs/heads/master
Commit: b04776d4d5235340148233f5f9e3f26f4c01076b
Parents: 10b3fee
Author: Pei He 
Authored: Wed Aug 10 15:21:41 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 13:44:15 2016 -0700

--
 .../common/ExampleBigQueryTableOptions.java |  3 +-
 .../beam/examples/common/ExampleOptions.java| 43 --
 ...xamplePubsubTopicAndSubscriptionOptions.java |  2 +-
 .../common/ExamplePubsubTopicOptions.java   |  2 +-
 .../runners/flink/FlinkPipelineOptions.java | 34 --
 .../options/DataflowPipelineOptions.java| 48 
 .../options/DataflowPipelineOptionsTest.java| 42 ++---
 .../beam/sdk/options/PipelineOptions.java   | 42 +
 8 files changed, 80 insertions(+), 136 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b04776d4/examples/java/src/main/java/org/apache/beam/examples/common/ExampleBigQueryTableOptions.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleBigQueryTableOptions.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleBigQueryTableOptions.java
index 2eef525..5d815c7 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleBigQueryTableOptions.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleBigQueryTableOptions.java
@@ -49,8 +49,7 @@ public interface ExampleBigQueryTableOptions extends 
GcpOptions {
   static class BigQueryTableFactory implements DefaultValueFactory {
 @Override
 public String create(PipelineOptions options) {
-  return options.as(ExampleOptions.class).getNormalizedUniqueName()
-  .replace('-', '_');
+  return options.getJobName().replace('-', '_');
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b04776d4/examples/java/src/main/java/org/apache/beam/examples/common/ExampleOptions.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleOptions.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleOptions.java
index 8b7ed07..85643e4 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleOptions.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleOptions.java
@@ -17,17 +17,9 @@
  */
 package org.apache.beam.examples.common;
 
-import com.google.common.base.MoreObjects;
-import java.util.concurrent.ThreadLocalRandom;
-import org.apache.beam.sdk.options.ApplicationNameOptions;
 import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.DefaultValueFactory;
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.joda.time.DateTimeUtils;
-import org.joda.time.DateTimeZone;
-import org.joda.time.format.DateTimeFormat;
-import org.joda.time.format.DateTimeFormatter;
 
 /**
  * Options that can be used to configure the Beam examples.
@@ -42,39 +34,4 @@ public interface ExampleOptions extends PipelineOptions {
   @Default.Integer(1)
   int getInjectorNumWorkers();
   void setInjectorNumWorkers(int numWorkers);
-
-  @Description("A normalized unique name that is used to name anything related 
to the pipeline."
-  + "It defaults to ApplicationName-UserName-Date-RandomInteger")
-  @Default.InstanceFactory(NormalizedUniqueNameFactory.class)
-  String getNormalizedUniqueName();
-  void setNormalizedUniqueName(String numWorkers);
-
-  /**
-   * Returns a normalized unique name constructed from {@link 
ApplicationNameOptions#getAppName()},
-   * the local system user name (if available), the current time, and a random 
integer.
-   *
-   * The normalization makes sure that the name matches the pattern of
-   * [a-z]([-a-z0-9]*[a-z0-9])?.
-   */
-  public static class NormalizedUniqueNameFactory implements 
DefaultValueFactory {
-private static final DateTimeFormatter FORMATTER =
-DateTimeFormat.forPattern("MMddHHmmss").withZone(DateTimeZone.UTC);
-
-@Override
-public String create(PipelineOptions options) {
-  String appName = options.as(ApplicationNameOptions.class).getAppName();
-  String normalizedAppName = appName == null || appName.length() == 0 ? 
"BeamApp"
-  : appName.toLowerCase()
-   

[1/2] incubator-beam git commit: Closes #719

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 10b3fee34 -> a69a0ea90


Closes #719


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a69a0ea9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a69a0ea9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a69a0ea9

Branch: refs/heads/master
Commit: a69a0ea90a9dfeb295b9cef0b2f22b3ec91aa2ba
Parents: 10b3fee b04776d
Author: Dan Halperin 
Authored: Thu Aug 25 13:44:15 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 13:44:15 2016 -0700

--
 .../common/ExampleBigQueryTableOptions.java |  3 +-
 .../beam/examples/common/ExampleOptions.java| 43 --
 ...xamplePubsubTopicAndSubscriptionOptions.java |  2 +-
 .../common/ExamplePubsubTopicOptions.java   |  2 +-
 .../runners/flink/FlinkPipelineOptions.java | 34 --
 .../options/DataflowPipelineOptions.java| 48 
 .../options/DataflowPipelineOptionsTest.java| 42 ++---
 .../beam/sdk/options/PipelineOptions.java   | 42 +
 8 files changed, 80 insertions(+), 136 deletions(-)
--

[1/2] incubator-beam git commit: Move the samples data to gs://apache-beam-samples/

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f7384e1a6 -> 10b3fee34


Move the samples data to gs://apache-beam-samples/


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/83ccf082
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/83ccf082
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/83ccf082

Branch: refs/heads/master
Commit: 83ccf082ceb129f536e8a01b755c15ec0f5c2495
Parents: f7384e1
Author: Pei He 
Authored: Wed Aug 24 17:26:50 2016 -0700
Committer: Pei He 
Committed: Wed Aug 24 17:57:07 2016 -0700

--
 .../main/java/org/apache/beam/examples/DebuggingWordCount.java  | 4 ++--
 .../main/java/org/apache/beam/examples/MinimalWordCount.java| 2 +-
 .../main/java/org/apache/beam/examples/WindowedWordCount.java   | 2 +-
 .../java/src/main/java/org/apache/beam/examples/WordCount.java  | 4 ++--
 .../org/apache/beam/examples/complete/StreamingWordExtract.java | 2 +-
 .../src/main/java/org/apache/beam/examples/complete/TfIdf.java  | 4 ++--
 .../org/apache/beam/examples/complete/TopWikipediaSessions.java | 5 +++--
 .../org/apache/beam/examples/complete/TrafficMaxLaneFlow.java   | 2 +-
 .../java/org/apache/beam/examples/complete/TrafficRoutes.java   | 2 +-
 .../org/apache/beam/examples/cookbook/DatastoreWordCount.java   | 2 +-
 .../java/org/apache/beam/examples/cookbook/DeDupExample.java| 4 ++--
 .../java/org/apache/beam/examples/cookbook/TriggerExample.java  | 2 +-
 .../java/org/apache/beam/examples/MinimalWordCountJava8.java| 2 +-
 .../org/apache/beam/examples/complete/game/HourlyTeamScore.java | 2 +-
 .../java/org/apache/beam/examples/complete/game/UserScore.java  | 2 +-
 .../org/apache/beam/examples/MinimalWordCountJava8Test.java | 2 +-
 16 files changed, 22 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/83ccf082/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
index 5a0930c..be3aa41 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
@@ -100,8 +100,8 @@ import org.slf4j.LoggerFactory;
  * that changing the default worker log level to TRACE or DEBUG will 
significantly increase
  * the amount of logs output.
  *
- * The input file defaults to {@code 
gs://dataflow-samples/shakespeare/kinglear.txt} and can be
- * overridden with {@code --inputFile}.
+ * The input file defaults to {@code 
gs://apache-beam-samples/shakespeare/kinglear.txt}
+ * and can be overridden with {@code --inputFile}.
  */
 public class DebuggingWordCount {
   /** A DoFn that filters for a specific key based upon a regular expression. 
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/83ccf082/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
index df725e3..f28a20c 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
@@ -87,7 +87,7 @@ public class MinimalWordCount {
 // Concept #1: Apply a root transform to the pipeline; in this case, 
TextIO.Read to read a set
 // of input text files. TextIO.Read returns a PCollection where each 
element is one line from
 // the input text (a set of Shakespeare's texts).
-p.apply(TextIO.Read.from("gs://dataflow-samples/shakespeare/*"))
+p.apply(TextIO.Read.from("gs://apache-beam-samples/shakespeare/*"))
  // Concept #2: Apply a ParDo transform to our PCollection of text lines. 
This ParDo invokes a
  // DoFn (defined in-line) on each element that tokenizes the text line 
into individual words.
  // The ParDo returns a PCollection, where each element is an 
individual word in

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/83ccf082/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
index 5f60524..7af354c 100644
--- 

[GitHub] incubator-beam pull request #882: Move the samples data to gs://apache-beam-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/882


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #882

[dhalperi]

Closes #882


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/10b3fee3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/10b3fee3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/10b3fee3

Branch: refs/heads/master
Commit: 10b3fee342878c1b5efac3e2ae956017d70db4a2
Parents: f7384e1 83ccf08
Author: Dan Halperin 
Authored: Thu Aug 25 13:02:59 2016 -0700
Committer: Dan Halperin 
Committed: Thu Aug 25 13:02:59 2016 -0700

--
 .../main/java/org/apache/beam/examples/DebuggingWordCount.java  | 4 ++--
 .../main/java/org/apache/beam/examples/MinimalWordCount.java| 2 +-
 .../main/java/org/apache/beam/examples/WindowedWordCount.java   | 2 +-
 .../java/src/main/java/org/apache/beam/examples/WordCount.java  | 4 ++--
 .../org/apache/beam/examples/complete/StreamingWordExtract.java | 2 +-
 .../src/main/java/org/apache/beam/examples/complete/TfIdf.java  | 4 ++--
 .../org/apache/beam/examples/complete/TopWikipediaSessions.java | 5 +++--
 .../org/apache/beam/examples/complete/TrafficMaxLaneFlow.java   | 2 +-
 .../java/org/apache/beam/examples/complete/TrafficRoutes.java   | 2 +-
 .../org/apache/beam/examples/cookbook/DatastoreWordCount.java   | 2 +-
 .../java/org/apache/beam/examples/cookbook/DeDupExample.java| 4 ++--
 .../java/org/apache/beam/examples/cookbook/TriggerExample.java  | 2 +-
 .../java/org/apache/beam/examples/MinimalWordCountJava8.java| 2 +-
 .../org/apache/beam/examples/complete/game/HourlyTeamScore.java | 2 +-
 .../java/org/apache/beam/examples/complete/game/UserScore.java  | 2 +-
 .../org/apache/beam/examples/MinimalWordCountJava8Test.java | 2 +-
 16 files changed, 22 insertions(+), 21 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #140

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make TransformEvaluatorFactory reuse Explicit

[dhalperi] Checkstyle: enforce package-info.java presence

[klk] Add withAllowedLateness with Closing Behavior to Window

[klk] Use the ParDo Application to Cache DoFns

[dhalperi] BigQueryIO: move to google-cloud-platform module

[klk] Add os-maven-plugin to Spark runner

[lcwik] BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

[lcwik] BigtableIO: upgrade to 0.9.1

[dhalperi] Rename DirectRegistrar Registrars

[dhalperi] [BEAM-316] Add file scheme support in TextIO

[dhalperi] [BEAM-316] Add file URI handling

[dhalperi] Execute RunnableOnService tests only when runner options provided

[jesse] Added BigInteger to TypeDescriptors class.

[dhalperi] [BEAM-480] Move insertAll() from BigQueryTableInserter to

[dhalperi] Remove dependency on beam-examples-java.

[dhalperi] Remove JavaDoc links to beam-examples.

[mxm] [flink] improve example section in README

[klk] Tidy WriteWithShardingFactory

[klk] Introduce StateInternalsFactory

[klk] Port runners to use GroupAlsoByWindows via StateInternalsFactory

[lcwik] Remove overrides of isStreaming() and getAppName() in

[owenzhang1990] [BEAM-489] remove headerLocation from maven-checkstyle-plugin

[dhalperi] Remove spark test dependency on DirectRunner

[dhalperi] Spark tests: force spark runner

[dhalperi] [BEAM-386] Remove StreamingCreate in DataflowRunner

[ansela] Reuse context and disable UI.

[owenzhang1990] [BEAM-328] uses SerializableCoder for `T extends Serializable`

[jbonofre] [BEAM-487] Update README.md: add DISCLAIMER, incubating, minor fixes

[dhalperi] CoderRegistry: make deprecated method private

[lcwik] Remove DataflowJUnitTestRunner as integration tests execute using

[jbonofre] [BEAM-488] Remove KEYS file

[jbonofre] [BEAM-486] Remove unnecessary mention of Apache v2.0 LICENSE

[dhalperi] [BEAM-13] Add JmsIO

[klk] Create DoFnInvoker instances in the package of the invoked DoFn

[dhalperi] [BEAM-433] Remove references to DataflowPipelineOptions

[dhalperi] [maven-release-plugin] prepare branch release-0.2.0-incubating

[dhalperi] [maven-release-plugin] prepare for next development iteration

[tgroh] Apply ExecutorUpdates in two Phases

[dhalperi] jms: fix artifact ID in pom.xml

[lcwik] Use AutoValue for StepTransformResult

[lcwik] Remove unneccessary method in ReadEvaluatorFactories

[bchambers] Implement Aggregators directly for direct-java

[dhalperi] [BEAM-443] Add waitToFinish() and cancel() in PipelineResult.

[klk] Move local Flink integration tests to a profile

[dhalperi] Added Bigtable Read DisplayData

[tgroh] Add handleEmpty to CompletionCallback

[dhalperi] Datastore Read as a composite PTransform

[dhalperi] Checkstyle: require assignment operator on same line

[bchambers] Add ElementByteSizeObserver default constructor

[klk] Name local Spark RunnableOnService profile more precisely

[dhalperi] DataflowRunner: test path validation

[dhalperi] Move DataflowPathValidator to sdk.util, rename to GcsPathValidator

[dhalperi] Move getPathValidator to GcsOptions and use it in temp/staging 
locations

[dhalperi] Some fixups to tests

[dhalperi] Miscellaneous fixups

[klk] Move test-jar back to default profile

[klk] Remove extraneous installation on Travis

[klk] Bind checkstyle:check to the test-compile phase to fail earlier

[dhalperi] Run integration tests in parallel

[klk] Add ProducedOutput method to TransformResult

[klk] Use the State of the Executor to drive progress

[klk] Make Coder.NonDeterministicException extend Exception instead of

[dhalperi] Rename DoFn to OldDoFn

[dhalperi] Rename NoOpDoFn to NoOpOldDoFn

[dhalperi] Rename DoFnWithContext to DoFn

[dhalperi] Port WordCount example from OldDoFn to DoFn

[dhalperi] Port MinimalWordCount example from OldDoFn to DoFn

[dhalperi] Port WindowedWordCount example from OldDoFn to DoFn

[dhalperi] Port DebuggingWordCount example from OldDoFn to DoFn

[dhalperi] Port AutoComplete example from OldDoFn to DoFn

[dhalperi] Port microbenchmarks to new vocabulary

[dhalperi] Update Dataflow runner to worker container supporting OldDoFn

[klk] Run findbugs in the test-compile phase

[dhalperi] SparkRunner calls pipeline.run

[dhalperi] [BEAM-475] Javadoc has invalid references after the recent 
refactorings

[dhalperi] [BEAM-383] Modified BigQueryIO to write based on number of files and

[klk] Exclude "uncallable method of anonymous class" errors from findbugs

[dhalperi] The new DoFn is not @Experimental

[dhalperi] Deprecate OldDoFn

[dhalperi] [BEAM-124] Flink Running WordCountIT Example

[dhalperi] add back unnecessary changes

[dhalperi] Port examples to new DoFn

[klk] Correctly determine if DoFn has an anonymous class in ParDo

[klk] Propagate getAllowedTimestampSkew from DoFn to its adapter

[klk] Port join library to new DoFn

[klk] Port mentions of OldDoFn in PipelineOptions

[klk] Port 

[33/50] incubator-beam git commit: This closes #807

[kenn]

This closes #807


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/95e7f017
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/95e7f017
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/95e7f017

Branch: refs/heads/gearpump-runner
Commit: 95e7f017d16878d185a806a327eb24cd22900393
Parents: eef26f4 20208d6
Author: Kenneth Knowles 
Authored: Wed Aug 10 10:00:56 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 10:00:56 2016 -0700

--
 .../org/apache/beam/sdk/transforms/DoFn.java|  44 +-
 .../beam/sdk/transforms/DoFnReflector.java  |  92 +++
 .../beam/sdk/transforms/DoFnReflectorTest.java  | 157 ++-
 .../transforms/DoFnReflectorBenchmark.java  |  13 +-
 4 files changed, 214 insertions(+), 92 deletions(-)
--

[28/50] incubator-beam git commit: Fixed javadoc for processBundle

[kenn]

Fixed javadoc for processBundle

The 2nd `li` element for `processBundle(InputT... inputElements)` was missing a 
proper closing tag.

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a9a76b01
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a9a76b01
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a9a76b01

Branch: refs/heads/gearpump-runner
Commit: a9a76b01204cde989d69ec4a79b96d430f821f6f
Parents: 6a5310a
Author: Stas Levin 
Authored: Wed Aug 10 15:20:50 2016 +0300
Committer: Dan Halperin 
Committed: Wed Aug 10 09:21:21 2016 -0700

--
 .../src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a9a76b01/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index e2764eb..93d7884 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -170,7 +170,7 @@ public class DoFnTester {
*
* 
*   Calls {@link #startBundle}.
-   *   Calls {@link #processElement} on each of the arguments.
+   *   Calls {@link #processElement} on each of the arguments.
*   Calls {@link #finishBundle}.
*   Returns the result of {@link #takeOutputElements}.
* 

[34/50] incubator-beam git commit: Use Datastore AncestorKey for AutoComplete example

[kenn]

Use Datastore AncestorKey for AutoComplete example


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a41dab86
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a41dab86
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a41dab86

Branch: refs/heads/gearpump-runner
Commit: a41dab8685c086f60d0cd72b61ce7affd3559a34
Parents: 95e7f01
Author: Vikas Kedigehalli 
Authored: Thu Aug 4 11:29:25 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 10 10:39:15 2016 -0700

--
 .../beam/examples/complete/AutoComplete.java| 21 +++-
 1 file changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a41dab86/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
index 1ab39c9..baae100 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
@@ -18,6 +18,7 @@
 package org.apache.beam.examples.complete;
 
 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.datastore.v1beta3.client.DatastoreHelper.makeKey;
 import static com.google.datastore.v1beta3.client.DatastoreHelper.makeValue;
 
 import org.apache.beam.examples.common.ExampleBigQueryTableOptions;
@@ -61,7 +62,6 @@ import com.google.common.base.MoreObjects;
 import com.google.datastore.v1beta3.Entity;
 import com.google.datastore.v1beta3.Key;
 import com.google.datastore.v1beta3.Value;
-import com.google.datastore.v1beta3.client.DatastoreHelper;
 
 import org.joda.time.Duration;
 
@@ -384,18 +384,23 @@ public class AutoComplete {
   /**
* Takes as input a the top candidates per prefix, and emits an entity
* suitable for writing to Datastore.
+   *
+   * Note: We use ancestor keys for strong consistency. See the Cloud 
Datastore documentation on
+   * https://cloud.google.com/datastore/docs/concepts/structuring_for_strong_consistency;>
+   * Structuring Data for Strong Consistency
*/
   static class FormatForDatastore extends DoFn, Entity> {
 private String kind;
-
-public FormatForDatastore(String kind) {
+private String ancestorKey;
+public FormatForDatastore(String kind, String ancestorKey) {
   this.kind = kind;
+  this.ancestorKey = ancestorKey;
 }
 
 @ProcessElement
 public void processElement(ProcessContext c) {
   Entity.Builder entityBuilder = Entity.newBuilder();
-  Key key = DatastoreHelper.makeKey(kind, c.element().getKey()).build();
+  Key key = makeKey(makeKey(kind, ancestorKey).build(), kind, 
c.element().getKey()).build();
 
   entityBuilder.setKey(key);
   List candidates = new ArrayList<>();
@@ -444,6 +449,11 @@ public class AutoComplete {
 Boolean getOutputToDatastore();
 void setOutputToDatastore(Boolean value);
 
+@Description("Datastore ancestor key")
+@Default.String("root")
+String getDatastoreAncestorKey();
+void setDatastoreAncestorKey(String value);
+
 @Description("Datastore output project ID, defaults to project ID")
 String getOutputProject();
 void setOutputProject(String value);
@@ -476,7 +486,8 @@ public class AutoComplete {
 
 if (options.getOutputToDatastore()) {
   toWrite
-  .apply("FormatForDatastore", ParDo.of(new 
FormatForDatastore(options.getKind(
+  .apply("FormatForDatastore", ParDo.of(new 
FormatForDatastore(options.getKind(),
+  options.getDatastoreAncestorKey(
   
.apply(DatastoreIO.v1beta3().write().withProjectId(MoreObjects.firstNonNull(
   options.getOutputProject(), options.getProject(;
 }

[24/50] incubator-beam git commit: Change WordCount default input

[kenn]

Change WordCount default input


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e979c826
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e979c826
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e979c826

Branch: refs/heads/gearpump-runner
Commit: e979c826873b61f55c296cc07262829d63610a99
Parents: 9983ff9
Author: Mark Liu 
Authored: Tue Aug 9 15:27:20 2016 -0700
Committer: Mark Liu 
Committed: Tue Aug 9 15:28:24 2016 -0700

--
 .../src/main/java/org/apache/beam/examples/WordCount.java | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e979c826/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
index d136b88..72ad35f 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
@@ -39,6 +39,7 @@ import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
 import com.google.common.base.Strings;
+import com.google.common.io.Resources;
 
 import java.io.IOException;
 
@@ -204,16 +205,16 @@ public class WordCount {
  *   SparkRunner:
  *   .../src/test/resources/LICENSE
  *   other runners:
- *   gs://dataflow-samples/apache/LICENSE
+ *   gs://apache-beam-samples/apache/LICENSE
  * 
  */
 public static class InputFactory implements DefaultValueFactory {
   @Override
   public String create(PipelineOptions options) {
 if (options.getRunner().isAssignableFrom(SparkRunner.class)) {
-  return WordCount.class.getClass().getResource("/LICENSE").getPath();
+  return Resources.getResource("LICENSE").getPath();
 } else {
-  return "gs://dataflow-samples/apache/LICENSE";
+  return "gs://apache-beam-samples/apache/LICENSE";
 }
   }
 }

[38/50] incubator-beam git commit: Move aggregator support classes out of runners namespace, make private

[kenn]

Move aggregator support classes out of runners namespace, make private


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/adec254d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/adec254d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/adec254d

Branch: refs/heads/gearpump-runner
Commit: adec254d5fdb409e786a1fc2bcee38f8a7a04408
Parents: 9da4bbc
Author: Kenneth Knowles 
Authored: Fri Jul 1 14:56:20 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 11:34:03 2016 -0700

--
 .../beam/runners/direct/DirectRunner.java   |   7 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../runners/dataflow/DataflowPipelineJob.java   |   4 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   4 +-
 .../dataflow/DataflowPipelineJobTest.java   |   4 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../spark/translation/SparkRuntimeContext.java  |   2 +-
 .../translation/MultiOutputWordCountTest.java   |   2 +-
 .../beam/sdk/AggregatorPipelineExtractor.java   |  93 
 .../beam/sdk/AggregatorRetrievalException.java  |  33 +++
 .../org/apache/beam/sdk/AggregatorValues.java   |  52 +
 .../main/java/org/apache/beam/sdk/Pipeline.java |  10 +
 .../org/apache/beam/sdk/PipelineResult.java |   2 -
 .../runners/AggregatorPipelineExtractor.java|  93 
 .../runners/AggregatorRetrievalException.java   |  33 ---
 .../beam/sdk/runners/AggregatorValues.java  |  52 -
 .../sdk/AggregatorPipelineExtractorTest.java| 229 +++
 .../AggregatorPipelineExtractorTest.java| 229 ---
 .../apache/beam/sdk/transforms/DoFnTest.java|   1 +
 .../apache/beam/sdk/transforms/OldDoFnTest.java |   3 +-
 20 files changed, 434 insertions(+), 427 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/adec254d/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index a9c8ecb..f2b781e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -20,15 +20,14 @@ package org.apache.beam.runners.direct;
 import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupByKeyOnly;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import org.apache.beam.runners.direct.ViewEvaluatorFactory.ViewOverrideFactory;
+import org.apache.beam.sdk.AggregatorRetrievalException;
+import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.Pipeline.PipelineExecutionException;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.io.Write;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.runners.AggregatorPipelineExtractor;
-import org.apache.beam.sdk.runners.AggregatorRetrievalException;
-import org.apache.beam.sdk.runners.AggregatorValues;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
@@ -244,7 +243,7 @@ public class DirectRunner
 executor.start(consumerTrackingVisitor.getRootTransforms());
 
 Map> aggregatorSteps =
-new AggregatorPipelineExtractor(pipeline).getAggregatorSteps();
+pipeline.getAggregatorSteps();
 DirectPipelineResult result =
 new DirectPipelineResult(executor, context, aggregatorSteps);
 if (options.isBlockOnRun()) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/adec254d/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
index cae0b2a..923d54c 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
@@ -18,8 +18,8 @@
 package org.apache.beam.runners.flink;
 
 import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.runners.AggregatorRetrievalException;
-import org.apache.beam.sdk.runners.AggregatorValues;

[12/50] incubator-beam git commit: Fix MapAggregatorValues use of toStringHelper

[kenn]

Fix MapAggregatorValues use of toStringHelper


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c867790a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c867790a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c867790a

Branch: refs/heads/gearpump-runner
Commit: c867790abad182bcbffa9a7a88de76a608b6d039
Parents: d60a0a0
Author: Kenneth Knowles 
Authored: Thu Jul 21 20:18:05 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 13:55:24 2016 -0700

--
 .../main/java/org/apache/beam/sdk/util/MapAggregatorValues.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c867790a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MapAggregatorValues.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MapAggregatorValues.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MapAggregatorValues.java
index fe02666..3d949ec 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MapAggregatorValues.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MapAggregatorValues.java
@@ -43,7 +43,7 @@ public class MapAggregatorValues extends 
AggregatorValues {
 
   @Override
   public String toString() {
-return MoreObjects.toStringHelper(MapAggregatorValues.class)
+return MoreObjects.toStringHelper(this)
 .add("stepValues", stepValues)
 .toString();
   }

[21/50] incubator-beam git commit: [BEAM-534] Fix dead links in README.md

[kenn]

[BEAM-534] Fix dead links in README.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0812fe65
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0812fe65
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0812fe65

Branch: refs/heads/gearpump-runner
Commit: 0812fe6551a187e07627b1331f0aa5a8978aa939
Parents: 7da1a3d
Author: Jean-Baptiste Onofré 
Authored: Thu Aug 4 23:18:42 2016 +0200
Committer: Frances Perry 
Committed: Tue Aug 9 14:50:48 2016 -0700

--
 README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0812fe65/README.md
--
diff --git a/README.md b/README.md
index e72aec2..3f03ea5 100644
--- a/README.md
+++ b/README.md
@@ -110,4 +110,5 @@ To get involved in Apache Beam:
 ## More Information
 
 * [Apache Beam](http://beam.incubator.apache.org)
-* [Getting Started with Apache 
Beam](http://beam.incubator.apache.org/getting_started/)
+* [Overview](http://beam.incubator.apache.org/use/beam-overview/)
+* [Quickstart](http://beam.incubator.apache.org/use/quickstart/)

[50/50] incubator-beam git commit: This closess #750

[kenn]

This closess #750


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/091a15a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/091a15a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/091a15a0

Branch: refs/heads/gearpump-runner
Commit: 091a15a07c7625ae3009cefaecece3a29a34c109
Parents: 02b2248 bc1b354
Author: Kenneth Knowles 
Authored: Thu Aug 25 11:40:03 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Aug 25 11:40:03 2016 -0700

--
 .travis.yml |   17 +-
 .travis/README.md   |   23 +
 .travis/settings.xml|   33 +
 .travis/test_wordcount.sh   |  125 +
 KEYS|  141 -
 NOTICE  |1 -
 README.md   |   25 +-
 examples/java/pom.xml   |   48 +-
 .../beam/examples/DebuggingWordCount.java   |2 +-
 .../apache/beam/examples/MinimalWordCount.java  |2 +-
 .../apache/beam/examples/WindowedWordCount.java |6 +-
 .../org/apache/beam/examples/WordCount.java |   26 +-
 .../beam/examples/common/ExampleUtils.java  |4 +-
 .../examples/common/PubsubFileInjector.java |   10 +-
 .../beam/examples/complete/AutoComplete.java|   37 +-
 .../examples/complete/StreamingWordExtract.java |   12 +-
 .../apache/beam/examples/complete/TfIdf.java|   12 +-
 .../examples/complete/TopWikipediaSessions.java |   29 +-
 .../examples/complete/TrafficMaxLaneFlow.java   |8 +-
 .../beam/examples/complete/TrafficRoutes.java   |   10 +-
 .../examples/cookbook/BigQueryTornadoes.java|6 +-
 .../cookbook/CombinePerKeyExamples.java |6 +-
 .../examples/cookbook/DatastoreWordCount.java   |9 +-
 .../beam/examples/cookbook/FilterExamples.java  |   10 +-
 .../beam/examples/cookbook/JoinExamples.java|   10 +-
 .../examples/cookbook/MaxPerKeyExamples.java|6 +-
 .../beam/examples/cookbook/TriggerExample.java  |   21 +-
 .../org/apache/beam/examples/WordCountIT.java   |2 +-
 .../org/apache/beam/examples/WordCountTest.java |3 +-
 .../examples/complete/AutoCompleteTest.java |2 +-
 .../examples/cookbook/TriggerExampleTest.java   |2 +-
 examples/java/src/test/resources/LICENSE|  202 ++
 examples/java8/pom.xml  |7 +-
 .../beam/examples/complete/game/GameStats.java  |   16 +-
 .../beam/examples/complete/game/UserScore.java  |2 +-
 .../complete/game/utils/WriteToBigQuery.java|   18 +-
 .../game/utils/WriteWindowedToBigQuery.java |   14 +-
 .../examples/complete/game/UserScoreTest.java   |2 +-
 examples/pom.xml|2 +-
 pom.xml |   72 +-
 runners/core-java/pom.xml   |2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   28 +-
 .../core/UnboundedReadFromBoundedSource.java|6 +-
 .../apache/beam/runners/core/package-info.java  |   22 +
 .../apache/beam/sdk/util/AssignWindowsDoFn.java |   10 +-
 .../org/apache/beam/sdk/util/DoFnRunner.java|   21 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|   82 +-
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  104 +-
 .../beam/sdk/util/GroupAlsoByWindowsDoFn.java   |   16 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   14 +-
 .../sdk/util/GroupByKeyViaGroupByKeyOnly.java   |   85 +-
 .../sdk/util/LateDataDroppingDoFnRunner.java|4 +-
 .../apache/beam/sdk/util/PaneInfoTracker.java   |1 -
 .../apache/beam/sdk/util/ReduceFnRunner.java|4 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |   19 +-
 .../org/apache/beam/sdk/util/WatermarkHold.java |1 -
 .../beam/sdk/util/common/package-info.java  |   20 +
 .../org/apache/beam/sdk/util/package-info.java  |   20 +
 .../UnboundedReadFromBoundedSourceTest.java |2 +-
 .../sdk/util/GroupAlsoByWindowsProperties.java  |   50 +-
 ...oupAlsoByWindowsViaOutputBufferDoFnTest.java |8 +-
 .../beam/sdk/util/ReduceFnRunnerTest.java   |1 +
 .../apache/beam/sdk/util/ReduceFnTester.java|1 +
 .../beam/sdk/util/SimpleDoFnRunnerTest.java |6 +-
 runners/direct-java/pom.xml |2 +-
 .../runners/direct/AggregatorContainer.java |  183 ++
 .../direct/BoundedReadEvaluatorFactory.java |   26 +-
 .../beam/runners/direct/CloningThreadLocal.java |   43 +
 .../beam/runners/direct/CommittedResult.java|   23 +-
 .../beam/runners/direct/CompletionCallback.java |8 +
 .../beam/runners/direct/DirectRegistrar.java|4 +-
 .../beam/runners/direct/DirectRunner.java   |   57 +-
 .../beam/runners/direct/EvaluationContext.java  |   57 +-
 

[42/50] incubator-beam git commit: This closes #813

[kenn]

This closes #813


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6da92ad9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6da92ad9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6da92ad9

Branch: refs/heads/gearpump-runner
Commit: 6da92ad94701efce2643969ba9158371f6527a80
Parents: b4c0bb9 9fe6ce2
Author: Kenneth Knowles 
Authored: Wed Aug 10 14:28:12 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 14:28:12 2016 -0700

--
 .../apache/beam/sdk/util/TimerInternals.java| 15 ++-
 .../beam/sdk/util/TimerInternalsTest.java   | 47 
 2 files changed, 61 insertions(+), 1 deletion(-)
--

[39/50] incubator-beam git commit: This closes #699

[kenn]

This closes #699


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fa0e395f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fa0e395f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fa0e395f

Branch: refs/heads/gearpump-runner
Commit: fa0e395f0ad3e874206066881a273869e75d96a3
Parents: fc87a0c adec254
Author: Kenneth Knowles 
Authored: Wed Aug 10 11:34:17 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 11:34:17 2016 -0700

--
 .../beam/runners/direct/DirectRunner.java   |  25 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../runners/dataflow/DataflowPipelineJob.java   |  21 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   4 +-
 .../dataflow/DataflowPipelineJobTest.java   |   4 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../spark/translation/SparkRuntimeContext.java  |   2 +-
 .../translation/MultiOutputWordCountTest.java   |   2 +-
 .../beam/sdk/AggregatorPipelineExtractor.java   |  93 
 .../beam/sdk/AggregatorRetrievalException.java  |  33 +++
 .../org/apache/beam/sdk/AggregatorValues.java   |  52 +
 .../main/java/org/apache/beam/sdk/Pipeline.java |  10 +
 .../org/apache/beam/sdk/PipelineResult.java |   2 -
 .../runners/AggregatorPipelineExtractor.java|  93 
 .../runners/AggregatorRetrievalException.java   |  33 ---
 .../beam/sdk/runners/AggregatorValues.java  |  52 -
 .../beam/sdk/util/MapAggregatorValues.java  |  50 
 .../sdk/AggregatorPipelineExtractorTest.java| 229 +++
 .../AggregatorPipelineExtractorTest.java| 229 ---
 .../apache/beam/sdk/transforms/DoFnTest.java|   1 +
 .../apache/beam/sdk/transforms/OldDoFnTest.java |   3 +-
 21 files changed, 464 insertions(+), 482 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa0e395f/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--

[15/50] incubator-beam git commit: Relocate Travis utilities to .travis

[kenn]

Relocate Travis utilities to .travis


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5e1185d6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5e1185d6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5e1185d6

Branch: refs/heads/gearpump-runner
Commit: 5e1185d67cea4112bb08251de72f513dc9b4b871
Parents: 7136616
Author: Kenneth Knowles 
Authored: Thu Aug 4 19:35:39 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 09:59:05 2016 -0700

--
 .travis.yml  |   2 +-
 .travis/README.md|  23 +++
 .travis/test_wordcount.sh| 125 ++
 testing/travis/README.md |  23 ---
 testing/travis/test_wordcount.sh | 125 --
 5 files changed, 149 insertions(+), 149 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e1185d6/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 4674bf3..656aba0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -53,7 +53,7 @@ install:
 
 script:
   - travis_retry mvn --settings .travis/settings.xml --batch-mode 
--update-snapshots $MAVEN_OVERRIDE verify
-  - travis_retry testing/travis/test_wordcount.sh
+  - travis_retry .travis/test_wordcount.sh
 
 cache:
   directories:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e1185d6/.travis/README.md
--
diff --git a/.travis/README.md b/.travis/README.md
new file mode 100644
index 000..e0c13f2
--- /dev/null
+++ b/.travis/README.md
@@ -0,0 +1,23 @@
+
+
+# Travis Scripts
+
+This directory contains scripts used for [Travis 
CI](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)
+testing.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e1185d6/.travis/test_wordcount.sh
--
diff --git a/.travis/test_wordcount.sh b/.travis/test_wordcount.sh
new file mode 100755
index 000..e059a35
--- /dev/null
+++ b/.travis/test_wordcount.sh
@@ -0,0 +1,125 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script runs WordCount example locally in a few different ways.
+# Specifically, all combinations of:
+#  a) using mvn exec, or java -cp with a bundled jar file;
+#  b) input filename with no directory component, with a relative directory, or
+# with an absolute directory; AND
+#  c) input filename containing wildcards or not.
+#
+# The one optional parameter is a path from the directory containing the script
+# to the directory containing the top-level (parent) pom.xml.  If no parameter
+# is provided, the script assumes that directory is equal to the directory
+# containing the script itself.
+#
+# The exit-code of the script indicates success or a failure.
+
+set -e
+set -o pipefail
+
+PASS=1
+VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate 
-Dexpression=project.version | grep -v '\[')
+JAR_FILE=examples/java/target/beam-examples-java-bundled-${VERSION}.jar
+
+function check_result_hash {
+  local name=$1
+  local outfile_prefix=$2
+  local expected=$3
+
+  local actual=$(LC_ALL=C sort $outfile_prefix-* | md5sum | awk '{print $1}' \
+|| LC_ALL=C sort $outfile_prefix-* | md5 -q) || exit 2  # OSX
+  if [[ "$actual" != "$expected" ]]
+  then
+echo "FAIL $name: Output hash mismatch.  Got $actual, expected $expected."
+PASS=""
+echo "head hexdump of actual:"
+head $outfile_prefix-* | hexdump -c
+  else
+echo "pass $name"
+# Output files are left behind in /tmp
+  fi
+}
+
+function get_outfile_prefix {
+  local name=$1
+  # NOTE: mktemp on OSX doesn't support --tmpdir
+  mktemp -u "/tmp/$name.out.XX"
+}
+
+function run_via_mvn {
+  local name=$1
+  local input=$2
+  local expected_hash=$3
+
+  local outfile_prefix="$(get_outfile_prefix 

[46/50] incubator-beam git commit: move integration tests to profile

[kenn]

move integration tests to profile


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1672b548
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1672b548
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1672b548

Branch: refs/heads/gearpump-runner
Commit: 1672b5483e029292816397248dc6fe63bf51f4af
Parents: 2a0ba61
Author: manuzhang 
Authored: Sat Jul 23 14:10:15 2016 +0800
Committer: Kenneth Knowles 
Committed: Thu Aug 25 11:37:17 2016 -0700

--
 runners/gearpump/pom.xml | 107 --
 1 file changed, 61 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1672b548/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index c725dae..85994ad 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -57,6 +57,67 @@
 
   
 
+  
+
+  local-runnable-on-service-tests
+  false
+  
+
+  
+org.apache.maven.plugins
+maven-surefire-plugin
+2.19.1
+
+  
+runnable-on-service-tests
+
+  
org.apache.beam.sdk.testing.RunnableOnService
+  none
+  true
+  
+
org.apache.beam:beam-sdks-java-core
+
org.apache.beam:beam-runners-java-core
+  
+  
+
+
+  org.apache.beam.sdk.io.BigQueryIOTest,
+  org.apache.beam.sdk.io.CountingInputTest,
+  org.apache.beam.sdk.io.CountingSourceTest,
+  org.apache.beam.sdk.testing.PAssertTest,
+  org.apache.beam.sdk.transforms.ApproximateUniqueTest,
+  org.apache.beam.sdk.transforms.CombineTest,
+  org.apache.beam.sdk.transforms.CombineFnsTest,
+  org.apache.beam.sdk.transforms.CountTest,
+  org.apache.beam.sdk.transforms.FlattenTest,
+  org.apache.beam.sdk.transforms.ParDoTest,
+  org.apache.beam.sdk.transforms.SampleTest,
+  org.apache.beam.sdk.transforms.ViewTest,
+  org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
+
+
+
+  org.apache.beam.sdk.transforms.windowing.WindowingTest,
+  org.apache.beam.sdk.util.ReshuffleTest
+
+  
+  
+
+  [
+  "--runner=TestGearpumpRunner",
+  "--streaming=true"
+  ]
+
+  
+
+  
+
+  
+
+  
+
+  
+
   
 
   org.apache.gearpump
@@ -228,52 +289,6 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-2.19.1
-
-  
-runnable-on-service-tests
-
-  org.apache.beam.sdk.testing.RunnableOnService
-  none
-  true
-  
-org.apache.beam:beam-sdks-java-core
-org.apache.beam:beam-runners-java-core
-  
-  
-
-
-  org.apache.beam.sdk.io.BigQueryIOTest,
-  org.apache.beam.sdk.io.CountingInputTest,
-  org.apache.beam.sdk.io.CountingSourceTest,
-  org.apache.beam.sdk.testing.PAssertTest,
-  org.apache.beam.sdk.transforms.ApproximateUniqueTest,
-  org.apache.beam.sdk.transforms.CombineTest,
-  org.apache.beam.sdk.transforms.CombineFnsTest,
-  org.apache.beam.sdk.transforms.CountTest,
-  org.apache.beam.sdk.transforms.FlattenTest,
-  org.apache.beam.sdk.transforms.ParDoTest,
-  org.apache.beam.sdk.transforms.SampleTest,
-  org.apache.beam.sdk.transforms.ViewTest,
-  org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
-
-
-
-  org.apache.beam.sdk.transforms.windowing.WindowingTest,
-  org.apache.beam.sdk.util.ReshuffleTest
-
-  
-  
-
-  [
-  "--runner=TestGearpumpRunner",
-  "--streaming=true"
- 

[06/50] incubator-beam git commit: Add ViewFn and port SDK to use it

[kenn]

Add ViewFn and port SDK to use it

This is a preliminary step towards the architecture at
https://s.apache.org/beam-side-input-1-pager

This separates the ViewFn part of each PCollectionView
class/transform, toward eliminating extraneous public
subclasses of PCollectionView and PTransform.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c376b45c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c376b45c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c376b45c

Branch: refs/heads/gearpump-runner
Commit: c376b45cac8568d7242d29725f4a9a701673df75
Parents: 2b5c6bc
Author: Kenneth Knowles 
Authored: Wed Jun 22 08:39:33 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 12:31:11 2016 -0700

--
 .../beam/runners/direct/SideInputContainer.java |   2 +-
 .../runners/direct/ViewEvaluatorFactory.java|   5 +-
 .../functions/FlinkProcessContext.java  |   2 +-
 .../functions/SideInputInitializer.java |   2 +-
 .../spark/translation/SparkProcessContext.java  |   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   2 +-
 .../org/apache/beam/sdk/transforms/ViewFn.java  |  45 
 .../beam/sdk/util/DirectSideInputReader.java|   4 +-
 .../apache/beam/sdk/util/PCollectionViews.java  | 228 +--
 .../apache/beam/sdk/values/PCollectionView.java |  29 ++-
 .../sdk/testing/PCollectionViewTesting.java |  35 +--
 11 files changed, 262 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c376b45c/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
index 7a19ed9..6458215 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
@@ -247,7 +247,7 @@ class SideInputContainer {
   @SuppressWarnings("unchecked") Iterable values =
   (Iterable) 
viewContents.getUnchecked(PCollectionViewWindow.of(view,
   window)).get();
-  return view.fromIterableInternal(values);
+  return view.getViewFn().apply(values);
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c376b45c/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
index 7a0b0f7..362e903 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
@@ -133,8 +133,9 @@ class ViewEvaluatorFactory implements 
TransformEvaluatorFactory {
   /**
* An in-process implementation of the {@link CreatePCollectionView} 
primitive.
*
-   * This implementation requires the input {@link PCollection} to be an 
iterable, which is provided
-   * to {@link PCollectionView#fromIterableInternal(Iterable)}.
+   * This implementation requires the input {@link PCollection} to be an 
iterable
+   * of {@code WindowedValue}, which is provided
+   * to {@link PCollectionView#getViewFn()} for conversion to {@link ViewT}.
*/
   public static final class WriteView
   extends PTransform, PCollectionView> 
{

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c376b45c/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContext.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContext.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContext.java
index 3954d1f..64b93c8 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContext.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContext.java
@@ -249,7 +249,7 @@ class FlinkProcessContext
 view.getTagInternal().getId(), new SideInputInitializer<>(view));
 ViewT result = 

[14/50] incubator-beam git commit: This closes #791

[kenn]

This closes #791


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7da1a3de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7da1a3de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7da1a3de

Branch: refs/heads/gearpump-runner
Commit: 7da1a3dee0089acb7c791ec6d06f6b498334970a
Parents: 5049011 630ae25
Author: Kenneth Knowles 
Authored: Tue Aug 9 09:50:54 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 09:50:54 2016 -0700

--
 .travis.yml | 14 ++
 1 file changed, 14 insertions(+)
--

[36/50] incubator-beam git commit: SparkRunner batch interval as a configuration instead of Beam Windows.

[kenn]

SparkRunner batch interval as a configuration instead of Beam Windows.

Add the batch interval to the pipeline options, default arbitrarily to 1000 
msec.

Pick-up the batch interval from pipeline options and remove 
StreamingWindowPipelineDetector.

Use SDK API to get the window function.

Update the README

Update streaming tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/08dca30a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/08dca30a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/08dca30a

Branch: refs/heads/gearpump-runner
Commit: 08dca30a38e11c13e8a4b2db1529f1306cc489b0
Parents: 95e7f01
Author: Sela 
Authored: Wed Aug 10 13:30:30 2016 +0300
Committer: Sela 
Committed: Wed Aug 10 20:42:17 2016 +0300

--
 runners/spark/README.md |   3 +-
 .../runners/spark/SparkPipelineOptions.java |   5 +
 .../apache/beam/runners/spark/SparkRunner.java  |  19 +---
 .../streaming/StreamingTransformTranslator.java |  10 +-
 .../StreamingWindowPipelineDetector.java| 102 ---
 .../streaming/FlattenStreamingTest.java |   1 +
 .../streaming/KafkaStreamingTest.java   |   2 +
 .../streaming/SimpleStreamingWordCountTest.java |   2 +
 8 files changed, 18 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/08dca30a/runners/spark/README.md
--
diff --git a/runners/spark/README.md b/runners/spark/README.md
index d2bfd3e..ef42fa7 100644
--- a/runners/spark/README.md
+++ b/runners/spark/README.md
@@ -63,8 +63,7 @@ The Spark runner provides support for batch processing of 
Beam bounded PCollecti
 ### Streaming
 
 The Spark runner currently provides partial support for stream processing of 
Beam unbounded PCollections as Spark 
[DStream](http://spark.apache.org/docs/latest/streaming-programming-guide.html#discretized-streams-dstreams)s.
  
-Current implementation of *Windowing* takes the first window size in the 
pipeline and treats it as the Spark "batch interval", while following windows 
will be treated as *Processing Time* windows.  
-Further work is required to provide full support for the Beam Model 
*event-time* and *out-of-order* stream processing.
+Currently, both *FixedWindows* and *SlidingWindows* are supported, but only 
with processing-time triggers and discarding pane.  
 
 ### issue tracking
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/08dca30a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 6ef3741..080ff19 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -39,4 +39,9 @@ public interface SparkPipelineOptions extends 
PipelineOptions, StreamingOptions,
   @Default.Long(-1)
   Long getTimeout();
   void setTimeout(Long batchInterval);
+
+  @Description("Batch interval for Spark streaming in milliseconds.")
+  @Default.Long(1000)
+  Long getBatchIntervalMillis();
+  void setBatchIntervalMillis(Long batchInterval);
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/08dca30a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index d994ec4..be50f70 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -26,7 +26,6 @@ import 
org.apache.beam.runners.spark.translation.SparkProcessContext;
 import org.apache.beam.runners.spark.translation.TransformTranslator;
 import 
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext;
 import 
org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator;
-import 
org.apache.beam.runners.spark.translation.streaming.StreamingWindowPipelineDetector;
 import org.apache.beam.runners.spark.util.SinglePrimitiveOutputPTransform;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.PipelineOptions;
@@ -145,24 +144,16 @@ public final class SparkRunner extends 
PipelineRunner {
   public EvaluationResult run(Pipeline pipeline) {

[04/50] incubator-beam git commit: Port Java 8 tests to new DoFn

[kenn]

Port Java 8 tests to new DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/879f18fd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/879f18fd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/879f18fd

Branch: refs/heads/gearpump-runner
Commit: 879f18fd1694b0540e3e695416566f24220ecb1e
Parents: 331f523
Author: Kenneth Knowles 
Authored: Fri Aug 5 12:12:12 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 11:35:17 2016 -0700

--
 .../apache/beam/sdk/transforms/WithTimestampsJava8Test.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/879f18fd/sdks/java/java8tests/src/test/java/org/apache/beam/sdk/transforms/WithTimestampsJava8Test.java
--
diff --git 
a/sdks/java/java8tests/src/test/java/org/apache/beam/sdk/transforms/WithTimestampsJava8Test.java
 
b/sdks/java/java8tests/src/test/java/org/apache/beam/sdk/transforms/WithTimestampsJava8Test.java
index 1141e88..03aa647 100644
--- 
a/sdks/java/java8tests/src/test/java/org/apache/beam/sdk/transforms/WithTimestampsJava8Test.java
+++ 
b/sdks/java/java8tests/src/test/java/org/apache/beam/sdk/transforms/WithTimestampsJava8Test.java
@@ -47,9 +47,9 @@ public class WithTimestampsJava8Test implements Serializable {
  .apply(WithTimestamps.of((String input) -> new 
Instant(Long.valueOf(yearTwoThousand;
 
 PCollection> timestampedVals =
-timestamped.apply(ParDo.of(new OldDoFn>() {
-  @Override
-  public void processElement(OldDoFn>.ProcessContext c)
+timestamped.apply(ParDo.of(new DoFn>() {
+  @ProcessElement
+  public void processElement(ProcessContext c)
   throws Exception {
 c.output(KV.of(c.element(), c.timestamp()));
   }

[19/50] incubator-beam git commit: Deprecate more uses of OldDoFn

[kenn]

Deprecate more uses of OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2c8eb429
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2c8eb429
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2c8eb429

Branch: refs/heads/gearpump-runner
Commit: 2c8eb429b6710c3a637dda3a707fcc887e16858a
Parents: 7da1a3d
Author: Kenneth Knowles 
Authored: Mon Aug 8 13:27:00 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 12:41:52 2016 -0700

--
 .../src/main/java/org/apache/beam/sdk/transforms/ParDo.java| 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c8eb429/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 91f6203..ca6d9b2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -516,7 +516,10 @@ public class ParDo {
* {@code PCollection}, inferred from the types of the argument
* {@code OldDoFn}. It is ready to be applied, or further
* properties can be set on it first.
+   *
+   * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
*/
+  @Deprecated
   public static  Bound of(OldDoFn fn) {
 return of(fn, fn.getClass());
   }
@@ -620,7 +623,10 @@ public class ParDo {
  * not modify this transform. The resulting {@link PTransform} is
  * sufficiently specified to be applied, but more properties can
  * still be specified.
+ *
+ * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
  */
+@Deprecated
 public  Bound of(OldDoFn fn) {
   return of(fn, fn.getClass());
 }

[31/50] incubator-beam git commit: This closes #789

[kenn]

This closes #789


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eef26f42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eef26f42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/eef26f42

Branch: refs/heads/gearpump-runner
Commit: eef26f4238fb72e1a852297d3dc6b558dc567530
Parents: 038eeca 743923b
Author: Kenneth Knowles 
Authored: Wed Aug 10 09:44:42 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 09:44:42 2016 -0700

--
 .../src/main/resources/beam/findbugs-filter.xml |  6 -
 .../apache/beam/sdk/coders/StandardCoder.java   | 14 ++-
 .../beam/sdk/coders/StandardCoderTest.java  | 25 
 3 files changed, 33 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/eef26f42/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--

[09/50] incubator-beam git commit: This closes #768

[kenn]

This closes #768


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d60a0a0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d60a0a0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d60a0a0e

Branch: refs/heads/gearpump-runner
Commit: d60a0a0e412f51e98f6f024dfc7ec4f21d1355a4
Parents: 574c377 2b6c873
Author: Kenneth Knowles 
Authored: Mon Aug 8 13:43:07 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 13:43:07 2016 -0700

--
 .travis.yml |  5 +
 pom.xml | 21 +++--
 2 files changed, 8 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d60a0a0e/pom.xml
--

[40/50] incubator-beam git commit: This closes #803

[kenn]

This closes #803


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b4c0bb9f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b4c0bb9f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b4c0bb9f

Branch: refs/heads/gearpump-runner
Commit: b4c0bb9fa9951e6cfbc6aa56edcb8235a81216b1
Parents: fa0e395 fcdd15b
Author: Kenneth Knowles 
Authored: Wed Aug 10 13:41:29 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 13:41:29 2016 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 92 ++--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 29 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  6 ++
 3 files changed, 77 insertions(+), 50 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b4c0bb9f/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --cc 
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 54d1923,4d34ec4..bea6264
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@@ -2370,11 -2370,10 +2368,11 @@@ public class DataflowRunner extends Pip
}
  
/**
-* A specialized {@link OldDoFn} for writing the contents of a {@link 
PCollection}
+* A specialized {@link DoFn} for writing the contents of a {@link 
PCollection}
 * to a streaming {@link PCollectionView} backend implementation.
 */
 -  private static class StreamingPCollectionViewWriterFn
 +  @Deprecated
 +  public static class StreamingPCollectionViewWriterFn
extends OldDoFn implements OldDoFn.RequiresWindowAccess {
  private final PCollectionView view;
  private final Coder dataCoder;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b4c0bb9f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--

[35/50] incubator-beam git commit: Closes #788

[kenn]

Closes #788


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0ab9495f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0ab9495f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0ab9495f

Branch: refs/heads/gearpump-runner
Commit: 0ab9495f88aa8719789213c01808fe852ad89a91
Parents: 95e7f01 a41dab8
Author: Dan Halperin 
Authored: Wed Aug 10 10:39:15 2016 -0700
Committer: Dan Halperin 
Committed: Wed Aug 10 10:39:15 2016 -0700

--
 .../beam/examples/complete/AutoComplete.java| 21 +++-
 1 file changed, 16 insertions(+), 5 deletions(-)
--

[47/50] incubator-beam git commit: add package-info.java

[kenn]

add package-info.java


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/276a2e10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/276a2e10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/276a2e10

Branch: refs/heads/gearpump-runner
Commit: 276a2e106aa1a573fc2eb2426b640f63cf68
Parents: 1672b54
Author: manuzhang 
Authored: Thu Jul 28 16:30:13 2016 +0800
Committer: Kenneth Knowles 
Committed: Thu Aug 25 11:37:27 2016 -0700

--
 .../runners/gearpump/examples/package-info.java | 22 
 .../beam/runners/gearpump/package-info.java | 22 
 .../translators/functions/package-info.java | 22 
 .../gearpump/translators/io/package-info.java   | 22 
 .../gearpump/translators/package-info.java  | 22 
 .../translators/utils/package-info.java | 22 
 6 files changed, 132 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/276a2e10/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/examples/package-info.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/examples/package-info.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/examples/package-info.java
new file mode 100644
index 000..a62a6c0
--- /dev/null
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/examples/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Examples showcase Beam application over Gearpump runner.
+ */
+package org.apache.beam.runners.gearpump.examples;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/276a2e10/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/package-info.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/package-info.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/package-info.java
new file mode 100644
index 000..5013616
--- /dev/null
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Internal implementation of the Beam runner for Apache Gearpump.
+ */
+package org.apache.beam.runners.gearpump;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/276a2e10/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/package-info.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/package-info.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/package-info.java
new file mode 100644
index 000..cba2363
--- /dev/null
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation 

[45/50] incubator-beam git commit: Merge branch master into gearpump-runner

[kenn]

Merge branch master into gearpump-runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a0ba61e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a0ba61e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a0ba61e

Branch: refs/heads/gearpump-runner
Commit: 2a0ba61e8507e1539115b583749a78f14d577bd8
Parents: 02b2248 2a1055d
Author: Kenneth Knowles 
Authored: Thu Aug 25 11:36:45 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Aug 25 11:36:45 2016 -0700

--
 .travis.yml |   17 +-
 .travis/README.md   |   23 +
 .travis/settings.xml|   33 +
 .travis/test_wordcount.sh   |  125 +
 KEYS|  141 -
 NOTICE  |1 -
 README.md   |   25 +-
 examples/java/pom.xml   |   48 +-
 .../beam/examples/DebuggingWordCount.java   |2 +-
 .../apache/beam/examples/MinimalWordCount.java  |2 +-
 .../apache/beam/examples/WindowedWordCount.java |6 +-
 .../org/apache/beam/examples/WordCount.java |   26 +-
 .../beam/examples/common/ExampleUtils.java  |4 +-
 .../examples/common/PubsubFileInjector.java |   10 +-
 .../beam/examples/complete/AutoComplete.java|   37 +-
 .../examples/complete/StreamingWordExtract.java |   12 +-
 .../apache/beam/examples/complete/TfIdf.java|   12 +-
 .../examples/complete/TopWikipediaSessions.java |   29 +-
 .../examples/complete/TrafficMaxLaneFlow.java   |8 +-
 .../beam/examples/complete/TrafficRoutes.java   |   10 +-
 .../examples/cookbook/BigQueryTornadoes.java|6 +-
 .../cookbook/CombinePerKeyExamples.java |6 +-
 .../examples/cookbook/DatastoreWordCount.java   |9 +-
 .../beam/examples/cookbook/FilterExamples.java  |   10 +-
 .../beam/examples/cookbook/JoinExamples.java|   10 +-
 .../examples/cookbook/MaxPerKeyExamples.java|6 +-
 .../beam/examples/cookbook/TriggerExample.java  |   21 +-
 .../org/apache/beam/examples/WordCountIT.java   |2 +-
 .../org/apache/beam/examples/WordCountTest.java |3 +-
 .../examples/complete/AutoCompleteTest.java |2 +-
 .../examples/cookbook/TriggerExampleTest.java   |2 +-
 examples/java/src/test/resources/LICENSE|  202 ++
 examples/java8/pom.xml  |7 +-
 .../beam/examples/complete/game/GameStats.java  |   16 +-
 .../beam/examples/complete/game/UserScore.java  |2 +-
 .../complete/game/utils/WriteToBigQuery.java|   18 +-
 .../game/utils/WriteWindowedToBigQuery.java |   14 +-
 .../examples/complete/game/UserScoreTest.java   |2 +-
 examples/pom.xml|2 +-
 pom.xml |   72 +-
 runners/core-java/pom.xml   |2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   28 +-
 .../core/UnboundedReadFromBoundedSource.java|6 +-
 .../apache/beam/runners/core/package-info.java  |   22 +
 .../apache/beam/sdk/util/AssignWindowsDoFn.java |   10 +-
 .../org/apache/beam/sdk/util/DoFnRunner.java|   21 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|   82 +-
 .../org/apache/beam/sdk/util/DoFnRunners.java   |  104 +-
 .../beam/sdk/util/GroupAlsoByWindowsDoFn.java   |   16 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   14 +-
 .../sdk/util/GroupByKeyViaGroupByKeyOnly.java   |   85 +-
 .../sdk/util/LateDataDroppingDoFnRunner.java|4 +-
 .../apache/beam/sdk/util/PaneInfoTracker.java   |1 -
 .../apache/beam/sdk/util/ReduceFnRunner.java|4 +-
 .../apache/beam/sdk/util/SimpleDoFnRunner.java  |   19 +-
 .../org/apache/beam/sdk/util/WatermarkHold.java |1 -
 .../beam/sdk/util/common/package-info.java  |   20 +
 .../org/apache/beam/sdk/util/package-info.java  |   20 +
 .../UnboundedReadFromBoundedSourceTest.java |2 +-
 .../sdk/util/GroupAlsoByWindowsProperties.java  |   50 +-
 ...oupAlsoByWindowsViaOutputBufferDoFnTest.java |8 +-
 .../beam/sdk/util/ReduceFnRunnerTest.java   |1 +
 .../apache/beam/sdk/util/ReduceFnTester.java|1 +
 .../beam/sdk/util/SimpleDoFnRunnerTest.java |6 +-
 runners/direct-java/pom.xml |2 +-
 .../runners/direct/AggregatorContainer.java |  183 ++
 .../direct/BoundedReadEvaluatorFactory.java |   26 +-
 .../beam/runners/direct/CloningThreadLocal.java |   43 +
 .../beam/runners/direct/CommittedResult.java|   23 +-
 .../beam/runners/direct/CompletionCallback.java |8 +
 .../beam/runners/direct/DirectRegistrar.java|4 +-
 .../beam/runners/direct/DirectRunner.java   |   57 +-
 .../beam/runners/direct/EvaluationContext.java  |   57 +-
 

[27/50] incubator-beam git commit: This closes #792

[kenn]

This closes #792


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a5310a5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a5310a5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a5310a5

Branch: refs/heads/gearpump-runner
Commit: 6a5310a560e67743873b7b046951f27ef3b67ed9
Parents: a035796 5e1185d
Author: Kenneth Knowles 
Authored: Wed Aug 10 09:20:52 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Aug 10 09:20:52 2016 -0700

--
 .travis.yml  |   4 +-
 .travis/README.md|  23 +++
 .travis/settings.xml |  33 +
 .travis/test_wordcount.sh| 125 ++
 testing/travis/README.md |  23 ---
 testing/travis/test_wordcount.sh | 125 --
 6 files changed, 183 insertions(+), 150 deletions(-)
--

[49/50] incubator-beam git commit: Rename DoFn to OldDoFn in Gearpump runner

[kenn]

Rename DoFn to OldDoFn in Gearpump runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bc1b3549
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bc1b3549
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bc1b3549

Branch: refs/heads/gearpump-runner
Commit: bc1b354949416db3b52c4f37c66968bdb86f0813
Parents: 40be715
Author: manuzhang 
Authored: Fri Aug 12 07:22:00 2016 +0800
Committer: Kenneth Knowles 
Committed: Thu Aug 25 11:38:08 2016 -0700

--
 .../gearpump/GearpumpPipelineResult.java| 23 ++--
 .../gearpump/GearpumpPipelineRunner.java|  6 ++---
 .../gearpump/examples/StreamingWordCount.java   |  6 ++---
 .../translators/ParDoBoundMultiTranslator.java  |  3 ++-
 .../translators/ParDoBoundTranslator.java   |  3 ++-
 .../translators/functions/DoFnFunction.java |  3 ++-
 .../translators/utils/GearpumpDoFnRunner.java   | 23 ++--
 7 files changed, 45 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bc1b3549/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
index bc27147..6184bc3 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
@@ -17,11 +17,14 @@
  */
 package org.apache.beam.runners.gearpump;
 
+import org.apache.beam.sdk.AggregatorRetrievalException;
+import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.runners.AggregatorRetrievalException;
-import org.apache.beam.sdk.runners.AggregatorValues;
 import org.apache.beam.sdk.transforms.Aggregator;
+import org.joda.time.Duration;
+
+import java.io.IOException;
 
 /**
  * Result of executing a {@link Pipeline} with Gearpump.
@@ -33,10 +36,26 @@ public class GearpumpPipelineResult implements 
PipelineResult {
   }
 
   @Override
+  public State cancel() throws IOException {
+return null;
+  }
+
+  @Override
+  public State waitUntilFinish(Duration duration) throws IOException, 
InterruptedException {
+return null;
+  }
+
+  @Override
+  public State waitUntilFinish() throws IOException, InterruptedException {
+return null;
+  }
+
+  @Override
   public  AggregatorValues getAggregatorValues(Aggregator 
aggregator)
   throws AggregatorRetrievalException {
 throw new AggregatorRetrievalException(
 "PipelineResult getAggregatorValues not supported in Gearpump 
pipeline",
 new UnsupportedOperationException());
   }
+
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bc1b3549/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineRunner.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineRunner.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineRunner.java
index 660d703..4182ee4 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineRunner.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineRunner.java
@@ -23,8 +23,8 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsValidator;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -151,7 +151,7 @@ public class GearpumpPipelineRunner extends 
PipelineRunner wrapped;
 
-public AssignWindowsAndSetStrategy(Window.Bound wrapped) {
+AssignWindowsAndSetStrategy(Window.Bound wrapped) {
   this.wrapped = wrapped;
 }
 
@@ -184,7 +184,7 @@ public class GearpumpPipelineRunner extends 
PipelineRunner extends DoFn {
+  private static class IdentityFn extends OldDoFn {
 @Override
 public void processElement(ProcessContext c) {
   c.output(c.element());

[23/50] incubator-beam git commit: This closes #805

[kenn]

This closes #805


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/063ff2f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/063ff2f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/063ff2f4

Branch: refs/heads/gearpump-runner
Commit: 063ff2f42290654cefe6c8bc4ea066d94f9aeff6
Parents: ac5242e d70a3a4
Author: Kenneth Knowles 
Authored: Tue Aug 9 15:05:40 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 15:05:40 2016 -0700

--
 .../org/apache/beam/runners/dataflow/DataflowRunner.java | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)
--

[01/50] incubator-beam git commit: Port various Spark runner tests to new DoFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/gearpump-runner 02b2248a5 -> 091a15a07


Port various Spark runner tests to new DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5df3583
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5df3583
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5df3583

Branch: refs/heads/gearpump-runner
Commit: f5df358320cfde6a1c4d012d4169af691f6a18e9
Parents: d6395e9
Author: Kenneth Knowles 
Authored: Fri Aug 5 12:31:07 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 11:35:17 2016 -0700

--
 .../apache/beam/runners/spark/TfIdfTest.java| 22 ++--
 .../spark/translation/CombinePerKeyTest.java|  6 +++---
 .../translation/MultiOutputWordCountTest.java   | 10 -
 .../spark/translation/SerializationTest.java| 10 -
 .../streaming/KafkaStreamingTest.java   |  6 +++---
 5 files changed, 27 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5df3583/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
index 074e6aa..17bf6dd 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
@@ -24,8 +24,8 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Keys;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.RemoveDuplicates;
@@ -101,8 +101,8 @@ public class TfIdfTest {
   // of the words in the document associated with that that URI.
   PCollection> uriToWords = uriToContent
   .apply("SplitWords", ParDo.of(
-  new OldDoFn, KV>() {
-@Override
+  new DoFn, KV>() {
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey();
   String line = c.element().getValue();
@@ -144,8 +144,8 @@ public class TfIdfTest {
   // by the URI key.
   PCollection>> uriToWordAndCount = 
uriAndWordToCount
   .apply("ShiftKeys", ParDo.of(
-  new OldDoFn, Long>, KV>>() {
-@Override
+  new DoFn, Long>, KV>>() 
{
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey().getKey();
   String word = c.element().getKey().getValue();
@@ -183,8 +183,8 @@ public class TfIdfTest {
   // divided by the total number of words in the document.
   PCollection>> wordToUriAndTf = 
uriToWordAndCountAndTotal
   .apply("ComputeTermFrequencies", ParDo.of(
-  new OldDoFn, KV>>() 
{
-@Override
+  new DoFn, KV>>() {
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey();
   Long wordTotal = 
c.element().getValue().getOnly(wordTotalsTag);
@@ -208,8 +208,8 @@ public class TfIdfTest {
   PCollection> wordToDf = wordToDocCount
   .apply("ComputeDocFrequencies", ParDo
   .withSideInputs(totalDocuments)
-  .of(new OldDoFn, KV>() {
-@Override
+  .of(new DoFn, KV>() {
+@ProcessElement
 public void processElement(ProcessContext c) {
   String word = c.element().getKey();
   Long documentCount = c.element().getValue();
@@ -237,8 +237,8 @@ public class TfIdfTest {
   // divided by the log of the document frequency.
   return wordToUriAndTfAndDf
   .apply("ComputeTfIdf", ParDo.of(
-  new OldDoFn, 

[43/50] incubator-beam git commit: Demonstrate that the DirectRunner runs per-call

[kenn]

Demonstrate that the DirectRunner runs per-call

Add a field that is modified per output, which should occur twice.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/65f9076d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/65f9076d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/65f9076d

Branch: refs/heads/gearpump-runner
Commit: 65f9076d654be02cbdc07442d008f6c5245d1ab5
Parents: 6da92ad
Author: Thomas Groh 
Authored: Wed Aug 10 11:29:38 2016 -0700
Committer: bchambers 
Committed: Wed Aug 10 15:54:09 2016 -0700

--
 .../apache/beam/runners/direct/DirectRunnerTest.java | 15 +++
 1 file changed, 15 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/65f9076d/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
index 1e73ec0..ddce458 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
+import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.fail;
@@ -32,6 +33,7 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.OldDoFn;
@@ -58,6 +60,7 @@ import java.io.Serializable;
 import java.util.Arrays;
 import java.util.List;
 import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
 
 /**
  * Tests for basic {@link DirectRunner} functionality.
@@ -109,10 +112,12 @@ public class DirectRunnerTest implements Serializable {
 result.awaitCompletion();
   }
 
+  private static AtomicInteger changed;
   @Test
   public void reusePipelineSucceeds() throws Throwable {
 Pipeline p = getPipeline();
 
+changed = new AtomicInteger(0);
 PCollection> counts =
 p.apply(Create.of("foo", "bar", "foo", "baz", "bar", "foo"))
 .apply(MapElements.via(new SimpleFunction() {
@@ -131,6 +136,14 @@ public class DirectRunnerTest implements Serializable {
   }
 }));
 
+counts.apply(ParDo.of(new DoFn, Void>() {
+  @ProcessElement
+  public void updateChanged(ProcessContext c) {
+changed.getAndIncrement();
+  }
+}));
+
+
 PAssert.that(countStrs).containsInAnyOrder("baz: 1", "bar: 2", "foo: 3");
 
 DirectPipelineResult result = ((DirectPipelineResult) p.run());
@@ -138,6 +151,8 @@ public class DirectRunnerTest implements Serializable {
 
 DirectPipelineResult otherResult = ((DirectPipelineResult) p.run());
 otherResult.awaitCompletion();
+
+assertThat("Each element should have been processed twice", changed.get(), 
equalTo(6));
   }
 
   @Test(timeout = 5000L)

[11/50] incubator-beam git commit: Inline MapAggregatorValues to remove dependencies

[kenn]

Inline MapAggregatorValues to remove dependencies

This class is trivial. Adding it to the public API of the SDK is
not desirable, since it is just for runners. Adding it to runners-core
would be OK but is really overkill for a glorified Map.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9da4bbcd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9da4bbcd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9da4bbcd

Branch: refs/heads/gearpump-runner
Commit: 9da4bbcdaf3c19ee5f78836b7cffaab947861a58
Parents: c867790
Author: Kenneth Knowles 
Authored: Thu Jul 21 20:24:17 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 13:55:24 2016 -0700

--
 .../beam/runners/direct/DirectRunner.java   | 18 +--
 .../runners/dataflow/DataflowPipelineJob.java   | 17 ++-
 .../beam/sdk/util/MapAggregatorValues.java  | 50 
 3 files changed, 30 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9da4bbcd/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 743c565..a9c8ecb 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -36,7 +36,6 @@ import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
-import org.apache.beam.sdk.util.MapAggregatorValues;
 import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -47,6 +46,7 @@ import org.apache.beam.sdk.values.PInput;
 import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.PValue;
 
+import com.google.common.base.MoreObjects;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
@@ -321,7 +321,7 @@ public class DirectRunner
 throws AggregatorRetrievalException {
   AggregatorContainer aggregators = 
evaluationContext.getAggregatorContainer();
   Collection steps = aggregatorSteps.get(aggregator);
-  Map stepValues = new HashMap<>();
+  final Map stepValues = new HashMap<>();
   for (AppliedPTransform transform : 
evaluationContext.getSteps()) {
 if (steps.contains(transform.getTransform())) {
   T aggregate = aggregators.getAggregate(
@@ -331,7 +331,19 @@ public class DirectRunner
   }
 }
   }
-  return new MapAggregatorValues<>(stepValues);
+  return new AggregatorValues() {
+@Override
+public Map getValuesAtSteps() {
+  return stepValues;
+}
+
+@Override
+public String toString() {
+  return MoreObjects.toStringHelper(this)
+  .add("stepValues", stepValues)
+  .toString();
+}
+  };
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9da4bbcd/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index 3194f7c..a6baa4f 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -29,7 +29,6 @@ import org.apache.beam.sdk.runners.AggregatorValues;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.util.AttemptAndTimeBoundedExponentialBackOff;
 import org.apache.beam.sdk.util.AttemptBoundedExponentialBackOff;
-import org.apache.beam.sdk.util.MapAggregatorValues;
 
 import com.google.api.client.googleapis.json.GoogleJsonResponseException;
 import com.google.api.client.util.BackOff;
@@ -41,6 +40,7 @@ import com.google.api.services.dataflow.model.JobMessage;
 import com.google.api.services.dataflow.model.JobMetrics;
 import 

[44/50] incubator-beam git commit: This closes #811

[kenn]

This closes #811


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a1055dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a1055dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a1055dd

Branch: refs/heads/gearpump-runner
Commit: 2a1055dd8e669ee9181bda3d944e377f7e33b5ea
Parents: 6da92ad 65f9076
Author: bchambers 
Authored: Wed Aug 10 15:54:36 2016 -0700
Committer: bchambers 
Committed: Wed Aug 10 15:54:36 2016 -0700

--
 .../apache/beam/runners/direct/DirectRunnerTest.java | 15 +++
 1 file changed, 15 insertions(+)
--

[16/50] incubator-beam git commit: Lower maven central timeouts on Travis-CI

[kenn]

Lower maven central timeouts on Travis-CI


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/71366165
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/71366165
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/71366165

Branch: refs/heads/gearpump-runner
Commit: 713661651befe72a844a181a33884d5f6c1e72e9
Parents: 7da1a3d
Author: Kenneth Knowles 
Authored: Thu Aug 4 15:18:13 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 09:59:05 2016 -0700

--
 .travis.yml  |  2 +-
 .travis/settings.xml | 33 +
 2 files changed, 34 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71366165/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 43ae7de..4674bf3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -52,7 +52,7 @@ install:
   - rm -rf "$HOME/.m2/repository/org/apache/beam"
 
 script:
-  - travis_retry mvn --batch-mode --update-snapshots $MAVEN_OVERRIDE verify
+  - travis_retry mvn --settings .travis/settings.xml --batch-mode 
--update-snapshots $MAVEN_OVERRIDE verify
   - travis_retry testing/travis/test_wordcount.sh
 
 cache:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71366165/.travis/settings.xml
--
diff --git a/.travis/settings.xml b/.travis/settings.xml
new file mode 100644
index 000..e086aec
--- /dev/null
+++ b/.travis/settings.xml
@@ -0,0 +1,33 @@
+
+
+
+  
+
+  central
+  
+
+  
+1000
+  
+
+5000
+  
+
+  
+

[26/50] incubator-beam git commit: [BEAM-124] Spark Running WordCountIT Example

[kenn]

[BEAM-124] Spark Running WordCountIT Example

This closes #769


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a035796c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a035796c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a035796c

Branch: refs/heads/gearpump-runner
Commit: a035796c2b189ad06a39260590b17057e1a1da2c
Parents: 063ff2f e979c82
Author: Luke Cwik 
Authored: Wed Aug 10 08:25:40 2016 -0700
Committer: Luke Cwik 
Committed: Wed Aug 10 08:25:40 2016 -0700

--
 examples/java/pom.xml   |  24 +++
 .../org/apache/beam/examples/WordCount.java |  24 ++-
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 examples/java/src/test/resources/LICENSE| 202 +++
 4 files changed, 250 insertions(+), 2 deletions(-)
--

[02/50] incubator-beam git commit: Port Flink fork of examples to new DoFn

[kenn]

Port Flink fork of examples to new DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/87313f1c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/87313f1c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/87313f1c

Branch: refs/heads/gearpump-runner
Commit: 87313f1c3d8cf874e04aaf528161478afa030f38
Parents: ae1f6d1
Author: Kenneth Knowles 
Authored: Fri Aug 5 12:24:24 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 11:35:17 2016 -0700

--
 .../beam/runners/flink/examples/TFIDF.java  | 28 +++
 .../beam/runners/flink/examples/WordCount.java  |  5 +--
 .../flink/examples/streaming/AutoComplete.java  | 37 ++--
 .../flink/examples/streaming/JoinExamples.java  | 14 
 .../examples/streaming/KafkaIOExamples.java |  7 ++--
 .../KafkaWindowedWordCountExample.java  | 10 +++---
 .../examples/streaming/WindowedWordCount.java   | 10 +++---
 7 files changed, 57 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87313f1c/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
index 716c8ad..4deca12 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
@@ -32,7 +32,7 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.Validation;
 import org.apache.beam.sdk.transforms.Count;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.Keys;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -230,10 +230,10 @@ public class TFIDF {
   // Create a collection of pairs mapping a URI to each
   // of the words in the document associated with that that URI.
   PCollection> uriToWords = uriToContent
-  .apply("SplitWords", ParDo.of(new OldDoFn, KV>() {
+  .apply("SplitWords", ParDo.of(new DoFn, KV>() {
 private static final long serialVersionUID = 0;
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey();
   String line = c.element().getValue();
@@ -275,10 +275,10 @@ public class TFIDF {
   // by the URI key.
   PCollection>> uriToWordAndCount = 
uriAndWordToCount
   .apply("ShiftKeys", ParDo.of(
-  new OldDoFn, Long>, KV>>() {
+  new DoFn, Long>, KV>>() 
{
 private static final long serialVersionUID = 0;
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey().getKey();
   String word = c.element().getKey().getValue();
@@ -316,10 +316,10 @@ public class TFIDF {
   // divided by the total number of words in the document.
   PCollection>> wordToUriAndTf = 
uriToWordAndCountAndTotal
   .apply("ComputeTermFrequencies", ParDo.of(
-  new OldDoFn, KV>>() 
{
+  new DoFn, KV>>() {
 private static final long serialVersionUID = 0;
 
-@Override
+@ProcessElement
 public void processElement(ProcessContext c) {
   URI uri = c.element().getKey();
   Long wordTotal = 
c.element().getValue().getOnly(wordTotalsTag);
@@ -339,14 +339,14 @@ public class TFIDF {
   // documents in which the word appears divided by the total
   // number of documents in the corpus. Note how the total number of
   // documents is passed as a side input; the same value is
-  // presented to each invocation of the OldDoFn.
+  // presented to each invocation of the DoFn.
   PCollection> wordToDf = wordToDocCount
   .apply("ComputeDocFrequencies", ParDo
   .withSideInputs(totalDocuments)
-  

[17/50] incubator-beam git commit: Port easy parts of DataflowRunner to new DoFn

[kenn]

Port easy parts of DataflowRunner to new DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fcdd15b8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fcdd15b8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fcdd15b8

Branch: refs/heads/gearpump-runner
Commit: fcdd15b81b93f87de0aa02bfb3b09740bc259c4c
Parents: a1d601a
Author: Kenneth Knowles 
Authored: Mon Aug 8 20:35:59 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 12:41:52 2016 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 92 ++--
 1 file changed, 45 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fcdd15b8/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index fadd9c7..4d34ec4 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -761,31 +761,30 @@ public class DataflowRunner extends 
PipelineRunner {
   PTransform {
 
 /**
- * A {@link OldDoFn} that for each element outputs a {@code KV} structure 
suitable for
+ * A {@link DoFn} that for each element outputs a {@code KV} structure 
suitable for
  * grouping by the hash of the window's byte representation and sorting 
the grouped values
  * using the window's byte representation.
  */
 @SystemDoFnInternal
 private static class UseWindowHashAsKeyAndWindowAsSortKeyDoFn
-extends OldDoFn>> implements
-OldDoFn.RequiresWindowAccess {
+extends DoFn>> {
 
   private final IsmRecordCoder ismCoderForHash;
   private UseWindowHashAsKeyAndWindowAsSortKeyDoFn(IsmRecordCoder 
ismCoderForHash) {
 this.ismCoderForHash = ismCoderForHash;
   }
 
-  @Override
-  public void processElement(ProcessContext c) throws Exception {
+  @ProcessElement
+  public void processElement(ProcessContext c, BoundedWindow 
untypedWindow) throws Exception {
 @SuppressWarnings("unchecked")
-W window = (W) c.window();
+W window = (W) untypedWindow;
 c.output(
 KV.of(ismCoderForHash.hash(ImmutableList.of(window)),
 KV.of(window,
 WindowedValue.of(
 c.element(),
 c.timestamp(),
-c.window(),
+window,
 c.pane();
   }
 }
@@ -828,14 +827,14 @@ public class DataflowRunner extends 
PipelineRunner {
   extends PTransform {
 
 /**
- * A {@link OldDoFn} that outputs {@link IsmRecord}s. These records are 
structured as follows:
+ * A {@link DoFn} that outputs {@link IsmRecord}s. These records are 
structured as follows:
  * 
  *   Key 1: Window
  *   Value: Windowed value
  * 
  */
 static class IsmRecordForSingularValuePerWindowDoFn
-extends OldDoFn>>,
+extends DoFn>>,
  IsmRecord> {
 
   private final Coder windowCoder;
@@ -843,7 +842,7 @@ public class DataflowRunner extends 
PipelineRunner {
 this.windowCoder = windowCoder;
   }
 
-  @Override
+  @ProcessElement
   public void processElement(ProcessContext c) throws Exception {
 Optional previousWindowStructuralValue = Optional.absent();
 T previousValue = null;
@@ -902,7 +901,7 @@ public class DataflowRunner extends 
PipelineRunner {
 applyForSingleton(
 DataflowRunner runner,
 PCollection input,
-OldDoFn>>,
+DoFn>>,
  IsmRecord> doFn,
 boolean hasDefault,
 FinalT defaultValue,
@@ -998,7 +997,7 @@ public class DataflowRunner extends 
PipelineRunner {
   static class BatchViewAsList
   extends PTransform> {
 /**
-   

[41/50] incubator-beam git commit: Make TimerData#compareTo consistent with equals

[kenn]

Make TimerData#compareTo consistent with equals

Timers are equal if the domain, timestamp, and namespace are equal.
Compare these values in compareTo. The ordering of TimerData that are
not in the same namespace or domain is arbitrary.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9fe6ce22
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9fe6ce22
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9fe6ce22

Branch: refs/heads/gearpump-runner
Commit: 9fe6ce22ca12b48704deb0e7cf3c583dff9b1870
Parents: fc87a0c
Author: Thomas Groh 
Authored: Wed Aug 10 13:52:14 2016 -0700
Committer: Thomas Groh 
Committed: Wed Aug 10 13:55:54 2016 -0700

--
 .../apache/beam/sdk/util/TimerInternals.java| 15 ++-
 .../beam/sdk/util/TimerInternalsTest.java   | 47 
 2 files changed, 61 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9fe6ce22/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
index 3212d64..eb49b9d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
@@ -30,6 +30,7 @@ import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 
 import com.google.common.base.MoreObjects;
+import com.google.common.collect.ComparisonChain;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
@@ -205,9 +206,21 @@ public interface TimerInternals {
   .toString();
 }
 
+/**
+ * {@inheritDoc}.
+ *
+ * The ordering of {@link TimerData} that are not in the same namespace 
or domain is
+ * arbitrary.
+ */
 @Override
 public int compareTo(TimerData o) {
-  return Long.compare(timestamp.getMillis(), o.getTimestamp().getMillis());
+  ComparisonChain chain =
+  ComparisonChain.start().compare(timestamp, 
o.getTimestamp()).compare(domain, o.domain);
+  if (chain.result() == 0) {
+// Obtaining the stringKey may be expensive; only do so if required
+chain = chain.compare(namespace.stringKey(), o.namespace.stringKey());
+  }
+  return chain.result();
 }
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9fe6ce22/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TimerInternalsTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TimerInternalsTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TimerInternalsTest.java
index baf911a..bc2930c 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TimerInternalsTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TimerInternalsTest.java
@@ -17,12 +17,18 @@
  */
 package org.apache.beam.sdk.util;
 
+import static org.hamcrest.Matchers.comparesEqualTo;
+import static org.hamcrest.Matchers.lessThan;
+import static org.hamcrest.Matchers.not;
+import static org.junit.Assert.assertThat;
+
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.testing.CoderProperties;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.TimerInternals.TimerDataCoder;
+import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 
 import org.joda.time.Instant;
@@ -50,4 +56,45 @@ public class TimerInternalsTest {
 windowCoder, new IntervalWindow(new Instant(0), new 
Instant(100))),
 new Instant(99), TimeDomain.PROCESSING_TIME));
   }
+
+  @Test
+  public void testCompareTo() {
+Instant firstTimestamp = new Instant(100);
+Instant secondTimestamp = new Instant(200);
+IntervalWindow firstWindow = new IntervalWindow(new Instant(0), 
firstTimestamp);
+IntervalWindow secondWindow =  new IntervalWindow(firstTimestamp, 
secondTimestamp);
+Coder windowCoder = IntervalWindow.getCoder();
+
+StateNamespace firstWindowNs = StateNamespaces.window(windowCoder, 
firstWindow);
+StateNamespace secondWindowNs = StateNamespaces.window(windowCoder, 
secondWindow);
+
+TimerData firstEventTime = TimerData.of(firstWindowNs, firstTimestamp, 
TimeDomain.EVENT_TIME);
+TimerData 

[03/50] incubator-beam git commit: Port Flink integration tests to new DoFn

[kenn]

Port Flink integration tests to new DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ae1f6d18
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ae1f6d18
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ae1f6d18

Branch: refs/heads/gearpump-runner
Commit: ae1f6d181ebe3c0bdffc35c833a6fdc858937d6c
Parents: 879f18f
Author: Kenneth Knowles 
Authored: Fri Aug 5 12:17:20 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 11:35:17 2016 -0700

--
 .../java/org/apache/beam/runners/flink/ReadSourceITCase.java | 6 +++---
 .../apache/beam/runners/flink/ReadSourceStreamingITCase.java | 8 +---
 2 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ae1f6d18/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceITCase.java
--
diff --git 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceITCase.java
 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceITCase.java
index ca70096..516c7ba 100644
--- 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceITCase.java
+++ 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceITCase.java
@@ -20,7 +20,7 @@ package org.apache.beam.runners.flink;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.CountingInput;
 import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.values.PCollection;
 
@@ -72,8 +72,8 @@ public class ReadSourceITCase extends JavaProgramTestBase {
 
 PCollection result = p
 .apply(CountingInput.upTo(10))
-.apply(ParDo.of(new OldDoFn() {
-  @Override
+.apply(ParDo.of(new DoFn() {
+  @ProcessElement
   public void processElement(ProcessContext c) throws Exception {
 c.output(c.element().toString());
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ae1f6d18/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingITCase.java
--
diff --git 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingITCase.java
 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingITCase.java
index bc69f34..ea58d0d 100644
--- 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingITCase.java
+++ 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingITCase.java
@@ -20,9 +20,11 @@ package org.apache.beam.runners.flink;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.CountingInput;
 import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+
 import com.google.common.base.Joiner;
+
 import org.apache.flink.streaming.util.StreamingProgramTestBase;
 
 /**
@@ -59,8 +61,8 @@ public class ReadSourceStreamingITCase extends 
StreamingProgramTestBase {
 
 p
   .apply(CountingInput.upTo(10))
-  .apply(ParDo.of(new OldDoFn() {
-  @Override
+  .apply(ParDo.of(new DoFn() {
+  @ProcessElement
   public void processElement(ProcessContext c) throws Exception {
 c.output(c.element().toString());
   }

[20/50] incubator-beam git commit: Make StreamingPCollectionViewWriterFn and its data public

[kenn]

Make StreamingPCollectionViewWriterFn and its data public

This enables a smooth migration path whereby the runner can
notice the class and extract these values, to run them
without OldDoFn.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d70a3a47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d70a3a47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d70a3a47

Branch: refs/heads/gearpump-runner
Commit: d70a3a47ee39fab8c4514a7bda4126827baf65a1
Parents: 7da1a3d
Author: Kenneth Knowles 
Authored: Tue Aug 9 14:36:28 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 14:36:28 2016 -0700

--
 .../org/apache/beam/runners/dataflow/DataflowRunner.java | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d70a3a47/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index fadd9c7..6da33fa 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -2375,7 +2375,8 @@ public class DataflowRunner extends 
PipelineRunner {
* A specialized {@link OldDoFn} for writing the contents of a {@link 
PCollection}
* to a streaming {@link PCollectionView} backend implementation.
*/
-  private static class StreamingPCollectionViewWriterFn
+  @Deprecated
+  public static class StreamingPCollectionViewWriterFn
   extends OldDoFn implements OldDoFn.RequiresWindowAccess {
 private final PCollectionView view;
 private final Coder dataCoder;
@@ -2390,6 +2391,14 @@ public class DataflowRunner extends 
PipelineRunner {
   this.dataCoder = dataCoder;
 }
 
+public PCollectionView getView() {
+  return view;
+}
+
+public Coder getDataCoder() {
+  return dataCoder;
+}
+
 @Override
 public void processElement(ProcessContext c) throws Exception {
   List output = new ArrayList<>();

[13/50] incubator-beam git commit: Cache .m2 directory on Travis-CI

[kenn]

Cache .m2 directory on Travis-CI


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/630ae250
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/630ae250
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/630ae250

Branch: refs/heads/gearpump-runner
Commit: 630ae25017764a83cfd6dd6c1ac3d8eae2abfbb4
Parents: 5049011
Author: Kenneth Knowles 
Authored: Thu Aug 4 15:12:03 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 14:41:19 2016 -0700

--
 .travis.yml | 14 ++
 1 file changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/630ae250/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index bef61ff..43ae7de 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,6 +47,20 @@ before_install:
   - if [ "$TRAVIS_OS_NAME" == "linux" ]; then jdk_switcher use "$CUSTOM_JDK"; 
fi
   - export BEAM_SUREFIRE_ARGLINE="-Xmx512m"
 
+install:
+  # Removing this here protects from inadvertent caching
+  - rm -rf "$HOME/.m2/repository/org/apache/beam"
+
 script:
   - travis_retry mvn --batch-mode --update-snapshots $MAVEN_OVERRIDE verify
   - travis_retry testing/travis/test_wordcount.sh
+
+cache:
+  directories:
+- $HOME/.m2/repository
+
+before_cache:
+  # Removing here increases cache hits (makes the above
+  # rm in `install` redundant unless our config has a bug,
+  # but it will be idempotent)
+  - rm -rf "$HOME/.m2/repository/org/apache/beam"

[18/50] incubator-beam git commit: Unwrap UserCodeException in DoFnTester

[kenn]

Unwrap UserCodeException in DoFnTester

The execution of user code and system code is intertwined in the OldDoFn
wrapper of DoFn. So DoFnTester will sometimes encounter a wrapped
UserCodeException where previously the exception would not have
been wrapped.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a1d601af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a1d601af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a1d601af

Branch: refs/heads/gearpump-runner
Commit: a1d601afd0b98bf6183b14a8bbd5e6b8bee0233c
Parents: 2c8eb42
Author: Kenneth Knowles 
Authored: Tue Aug 9 12:39:41 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 9 12:41:52 2016 -0700

--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 29 ++--
 1 file changed, 26 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a1d601af/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index e2764eb..a2ce6c9 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -26,6 +26,7 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.TimerInternals;
+import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.InMemoryStateInternals;
@@ -190,10 +191,24 @@ public class DoFnTester {
 initializeState();
 TestContext context = createContext(fn);
 context.setupDelegateAggregators();
-fn.startBundle(context);
+try {
+  fn.startBundle(context);
+} catch (UserCodeException e) {
+  unwrapUserCodeException(e);
+}
 state = State.STARTED;
   }
 
+  private static void unwrapUserCodeException(UserCodeException e) throws 
Exception {
+if (e.getCause() instanceof Exception) {
+  throw (Exception) e.getCause();
+} else if (e.getCause() instanceof Error) {
+  throw (Error) e.getCause();
+} else {
+  throw e;
+}
+  }
+
   /**
* Calls {@link OldDoFn#processElement} on the {@code OldDoFn} under test, 
in a
* context where {@link OldDoFn.ProcessContext#element} returns the
@@ -212,7 +227,11 @@ public class DoFnTester {
 if (state == State.UNSTARTED) {
   startBundle();
 }
-fn.processElement(createProcessContext(fn, element));
+try {
+  fn.processElement(createProcessContext(fn, element));
+} catch (UserCodeException e) {
+  unwrapUserCodeException(e);
+}
   }
 
   /**
@@ -231,7 +250,11 @@ public class DoFnTester {
 if (state == State.UNSTARTED) {
   startBundle();
 }
-fn.finishBundle(createContext(fn));
+try {
+  fn.finishBundle(createContext(fn));
+} catch (UserCodeException e) {
+  unwrapUserCodeException(e);
+}
 state = State.FINISHED;
   }
 

[07/50] incubator-beam git commit: This closes #759

[kenn]

This closes #759


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bb00810a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bb00810a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bb00810a

Branch: refs/heads/gearpump-runner
Commit: bb00810adeee27506a43487c6842bb0c9dcf58a9
Parents: 7629f97 4753f44
Author: Kenneth Knowles 
Authored: Mon Aug 8 13:37:03 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Aug 8 13:37:03 2016 -0700

--
 .../beam/sdk/transforms/FlatMapElements.java |  2 +-
 .../apache/beam/sdk/transforms/MapElements.java  |  2 +-
 .../beam/sdk/transforms/SimpleFunction.java  | 14 +-
 .../beam/sdk/transforms/FlatMapElementsTest.java | 19 +++
 .../beam/sdk/transforms/MapElementsTest.java |  8 
 5 files changed, 42 insertions(+), 3 deletions(-)
--

[jira] [Created] (BEAM-588) All runners should support ProfilingOptions

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-588:


 Summary: All runners should support ProfilingOptions
 Key: BEAM-588
 URL: https://issues.apache.org/jira/browse/BEAM-588
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor


https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py#L366

This is useful for profiling pipelines in different environments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-587) Update KafkaIO to take advantage of Dofn Setup/Teardown

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-587:


 Summary: Update KafkaIO to take advantage of Dofn Setup/Teardown
 Key: BEAM-587
 URL: https://issues.apache.org/jira/browse/BEAM-587
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Daniel Halperin
Assignee: Raghu Angadi


Now that [~tgroh] has implemented DoFn setup/teardown in all runners, we should 
make KafkaIO/write's pardo only create the producer once in setup rather than 
in startBundle. Should be a huge efficiency win!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-586) CombineFns with no inputs should produce no outputs when used as a main input

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-586:


 Summary: CombineFns with no inputs should produce no outputs when 
used as a main input
 Key: BEAM-586
 URL: https://issues.apache.org/jira/browse/BEAM-586
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


TestGloballyEmptyCollection seems to be violating this assumption

https://github.com/apache/incubator-beam/pull/862/files#diff-a305819710e8d79d2b045d6416184f21R65
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-294) Change properties named dataflow.* to beam.*

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436761#comment-15436761
 ] 

ASF GitHub Bot commented on BEAM-294:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/884

[BEAM-294] Rename dataflow references to beam

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/884.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #884


commit 229c042a6b179236230fdfa61d5c0550a28e0d3b
Author: Jean-Baptiste Onofré 
Date:   2016-08-25T12:32:20Z

[BEAM-294] Rename dataflow references to beam




> Change properties named dataflow.* to beam.*
> 
>
> Key: BEAM-294
> URL: https://issues.apache.org/jira/browse/BEAM-294
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #884: [BEAM-294] Rename dataflow references to b...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/884

[BEAM-294] Rename dataflow references to beam

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/884.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #884


commit 229c042a6b179236230fdfa61d5c0550a28e0d3b
Author: Jean-Baptiste Onofré 
Date:   2016-08-25T12:32:20Z

[BEAM-294] Rename dataflow references to beam




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-286) Reorganize flink runner directories

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436708#comment-15436708
 ] 

Aljoscha Krettek commented on BEAM-286:
---

Now that the PR is in I think the reorganization/cleanup can be performed.

> Reorganize flink runner directories
> ---
>
> Key: BEAM-286
> URL: https://issues.apache.org/jira/browse/BEAM-286
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> The flink runner Maven module uses two sub-modules: runner and examples. It's 
> the only one which use such layout (compare to spark, dataflow or 
> inprocess/direct runners).
> I will propose a PR to align flink runner module with the other, keeping the 
> examples in a sub-directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #883: Address comments of Flink Side-Input PR

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/883

Address comments of Flink Side-Input PR

R: @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam flink-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/883.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #883


commit 5dbb6189c98e700121ea80f9249130033402ae07
Author: Aljoscha Krettek 
Date:   2016-08-25T09:00:39Z

Address comments of Flink Side-Input PR




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-286) Reorganize flink runner directories

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-286.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Reorganize flink runner directories
> ---
>
> Key: BEAM-286
> URL: https://issues.apache.org/jira/browse/BEAM-286
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> The flink runner Maven module uses two sub-modules: runner and examples. It's 
> the only one which use such layout (compare to spark, dataflow or 
> inprocess/direct runners).
> I will propose a PR to align flink runner module with the other, keeping the 
> examples in a sub-directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-286) Reorganize flink runner directories

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436459#comment-15436459
 ] 

Maximilian Michels commented on BEAM-286:
-

+1

> Reorganize flink runner directories
> ---
>
> Key: BEAM-286
> URL: https://issues.apache.org/jira/browse/BEAM-286
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The flink runner Maven module uses two sub-modules: runner and examples. It's 
> the only one which use such layout (compare to spark, dataflow or 
> inprocess/direct runners).
> I will propose a PR to align flink runner module with the other, keeping the 
> examples in a sub-directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-585) Support other message than TextMessage in JmsIO and improve custom coder support

[JIRA]

Jean-Baptiste Onofré created BEAM-585:
-

 Summary: Support other message than TextMessage in JmsIO and 
improve custom coder support
 Key: BEAM-585
 URL: https://issues.apache.org/jira/browse/BEAM-585
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


Now, the JmsIO only supports TextMessage coming from the broker. JMS broker can 
provide other kind of messages like BytesMessage, MapMessage, StreamMessage, 
ObjectMessage. The JmsIO should be able to deal with such kind of messages 
(implicitly and explicitly). Depending of the JMS message type, the JMS IO 
should use different coders.

On the other hand, the JmsIO should allow users to deal with their own custom 
coders.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-584) Support Verifiers in TestSparkRunner

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-584:
--

 Summary: Support Verifiers in TestSparkRunner
 Key: BEAM-584
 URL: https://issues.apache.org/jira/browse/BEAM-584
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Aviem Zur
Assignee: Amit Sela


[~jasonkuster] suggested that we should support verifiers to better support E2E 
tests.
See 
https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
 for an example of how they're used and 
https://github.com/apache/incubator-beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
 for how they are implemented in the TestDataflowRunner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)