[1/2] beam git commit: [BEAM-2412] Update HBaseIO to use HBase client 1.2.6

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master 32d55323c -> fe3d55403


[BEAM-2412] Update HBaseIO to use HBase client 1.2.6


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6859f804
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6859f804
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6859f804

Branch: refs/heads/master
Commit: 6859f80400bb16bbca34ed282e7a5e8a1328f955
Parents: 32d5532
Author: Ismaël Mejía 
Authored: Mon Jun 5 23:20:27 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue Jun 13 07:55:10 2017 +0200

--
 sdks/java/io/hbase/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6859f804/sdks/java/io/hbase/pom.xml
--
diff --git a/sdks/java/io/hbase/pom.xml b/sdks/java/io/hbase/pom.xml
index 746b993..f81cd24 100644
--- a/sdks/java/io/hbase/pom.xml
+++ b/sdks/java/io/hbase/pom.xml
@@ -31,7 +31,7 @@
   Library to read and write from/to HBase
 
   
-1.2.5
+1.2.6
 2.5.1
   
 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3350

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2423) Abstract StateInternalsTest for the different state internals/Runners

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047376#comment-16047376
 ] 

ASF GitHub Bot commented on BEAM-2423:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/3348

[BEAM-2423] Port state internals tests to the new base class 
StateInternalsTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam PortStateInternalsTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3348


commit 00ec33b98f4e041c88b4ae3fd609825405a69fd2
Author: JingsongLi 
Date:   2017-06-13T03:26:38Z

[BEAM-2423] Port state internals tests to the new base class 
StateInternalsTest




> Abstract StateInternalsTest for the different state internals/Runners
> -
>
> Key: BEAM-2423
> URL: https://issues.apache.org/jira/browse/BEAM-2423
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> For the test of InMemoryStateInternals, ApexStateInternals, 
> FlinkStateInternals, SparkStateInternals, etc..
> Have a common base class for the state internals test that has an abstract 
> method createStateInternals() and all the test methods. Then an actual 
> implementation would just derive from that and only implement the method for 
> creating the state internals. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3348: [BEAM-2423] Port state internals tests to the new b...

[JingsongLi]

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/3348

[BEAM-2423] Port state internals tests to the new base class 
StateInternalsTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam PortStateInternalsTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3348


commit 00ec33b98f4e041c88b4ae3fd609825405a69fd2
Author: JingsongLi 
Date:   2017-06-13T03:26:38Z

[BEAM-2423] Port state internals tests to the new base class 
StateInternalsTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Check for Deferral on Non-additional inputs

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master ce3dd4583 -> 32d55323c


Check for Deferral on Non-additional inputs

Because Side Inputs are represented within the expanded inputs, the
check that the transform is a Combine with Side Inputs would never be
hit. This ensures that we do not consider additional inputs during the
check to defer evaluation of the node.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1ac18b2e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1ac18b2e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1ac18b2e

Branch: refs/heads/master
Commit: 1ac18b2eb1371422e60d50a8c3f37b3b24d59611
Parents: ce3dd45
Author: Thomas Groh 
Authored: Mon Jun 12 16:55:59 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jun 12 16:55:59 2017 -0700

--
 .../main/java/org/apache/beam/runners/spark/SparkRunner.java  | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1ac18b2e/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index 9e2426e..d008718 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -26,6 +26,7 @@ import java.util.Map;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
+import org.apache.beam.runners.core.construction.TransformInputs;
 import org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator;
 import org.apache.beam.runners.spark.io.CreateStream;
 import org.apache.beam.runners.spark.metrics.AggregatorMetricSource;
@@ -359,10 +360,12 @@ public final class SparkRunner extends 
PipelineRunner {
 
 protected boolean shouldDefer(TransformHierarchy.Node node) {
   // if the input is not a PCollection, or it is but with non merging 
windows, don't defer.
-  if (node.getInputs().size() != 1) {
+  Collection nonAdditionalInputs =
+  
TransformInputs.nonAdditionalInputs(node.toAppliedPTransform(getPipeline()));
+  if (nonAdditionalInputs.size() != 1) {
 return false;
   }
-  PValue input = Iterables.getOnlyElement(node.getInputs().values());
+  PValue input = Iterables.getOnlyElement(nonAdditionalInputs);
   if (!(input instanceof PCollection)
   || ((PCollection) 
input).getWindowingStrategy().getWindowFn().isNonMerging()) {
 return false;

[2/2] beam git commit: This closes #3347: Check for Deferral on Non-additional inputs

[kenn]

This closes #3347: Check for Deferral on Non-additional inputs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/32d55323
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/32d55323
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/32d55323

Branch: refs/heads/master
Commit: 32d55323cd5a88f604ad9c763ab265de845df08c
Parents: ce3dd45 1ac18b2
Author: Kenneth Knowles 
Authored: Mon Jun 12 20:20:40 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Jun 12 20:20:40 2017 -0700

--
 .../main/java/org/apache/beam/runners/spark/SparkRunner.java  | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--

[jira] [Closed] (BEAM-2375) upgrade beam-version to 2.0.0

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu closed BEAM-2375.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2196) support UDF

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu closed BEAM-2196.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> support UDF
> ---
>
> Key: BEAM-2196
> URL: https://issues.apache.org/jira/browse/BEAM-2196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3349

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2194) JOIN: cross join

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2194:


Assignee: James Xu  (was: Xu Mingmin)

> JOIN: cross join
> 
>
> Key: BEAM-2194
> URL: https://issues.apache.org/jira/browse/BEAM-2194
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2433) unify SQL/Sql words in class name

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2433.

   Resolution: Fixed
Fix Version/s: Not applicable

> unify SQL/Sql words in class name
> -
>
> Key: BEAM-2433
> URL: https://issues.apache.org/jira/browse/BEAM-2433
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> Use {{Sql}} for class name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2289) add unit tests for BeamSQLFilterFn and BeamSQLProjectFn

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2289:
-
Labels: dsl_sql_merge  (was: )

> add unit tests for BeamSQLFilterFn and BeamSQLProjectFn
> ---
>
> Key: BEAM-2289
> URL: https://issues.apache.org/jira/browse/BEAM-2289
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Priority: Minor
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2196) support UDF

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047298#comment-16047298
 ] 

Xu Mingmin commented on BEAM-2196:
--

[~xumingming] can you close this task?

> support UDF
> ---
>
> Key: BEAM-2196
> URL: https://issues.apache.org/jira/browse/BEAM-2196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2281) call SqlFunctions in operator implementation

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2281:
-
Labels: dsl_sql_merge  (was: )

> call SqlFunctions in operator implementation
> 
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>  Labels: dsl_sql_merge
>
> Calcite has a collections of functions in 
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to 
> leverage when adding operators as {{BeamSqlExpression}}. 
> [~xumingming] [~app-tarush], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2282) logic in function accept in BeamSqlExpression

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2282:
-
Labels: dsl_sql_merge  (was: )

> logic in function accept in BeamSqlExpression
> -
>
> Key: BEAM-2282
> URL: https://issues.apache.org/jira/browse/BEAM-2282
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>  Labels: dsl_sql_merge
>
> Method {{accept}} is used to make sure the SQL operator is supported in Beam 
> SQL. IMO it should only check the case that some scenarios are not supported 
> in BeamSQL even it's valid in Calcite SQL grammar.
> It's not necessary to my understand to validate SQL grammar as that's done 
> already in the parsing phase.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2227) add a post to explain how Beam SQL works

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2227:
-
Labels: dsl_sql_merge  (was: )

> add a post to explain how Beam SQL works
> 
>
> Key: BEAM-2227
> URL: https://issues.apache.org/jira/browse/BEAM-2227
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> As mentioned in maillist, it's important to clarify what Beam SQL does, and 
> how it works.
> {quote}
> There's some confusion where people think we're just doing a pass through
> to the framework's SQL engine. We'll have to make sure we're clear on how
> Beam's SQL works in the docs.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2375) upgrade beam-version to 2.0.0

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047295#comment-16047295
 ] 

Xu Mingmin commented on BEAM-2375:
--

[~xumingming] can you close this task?

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2424) CAST operator supporting numeric types

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2424:
-
Labels: dsl_sql_merge  (was: )

> CAST operator supporting numeric types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2435) test BeamRelNode with DSL methods

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2435:
-
Labels: dsl_sql_merge  (was: )

> test BeamRelNode with DSL methods
> -
>
> Key: BEAM-2435
> URL: https://issues.apache.org/jira/browse/BEAM-2435
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> Test supported SQL operators in BeamRelNode, using DSL methods. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #78

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2343

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #126

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] [BEAM-2423] Abstract StateInternalsTest for the different 
state

[altay] Reverse removal of NativeWrite evaluator in Python DirectRunner

[tgroh] Cleanup Combine Tests with Context

--
[...truncated 2.65 MB...]
2017-06-13T00:35:23.726 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-13T00:35:23.737 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 1023.3 KB/sec)
2017-06-13T00:35:23.741 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-13T00:35:23.750 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1426.0 KB/sec)
2017-06-13T00:35:23.754 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-13T00:35:23.762 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 271.6 KB/sec)
2017-06-13T00:35:23.766 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-13T00:35:23.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 143.0 KB/sec)
2017-06-13T00:35:23.781 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-13T00:35:23.790 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 583.3 KB/sec)
2017-06-13T00:35:23.793 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-13T00:35:23.802 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 277.2 KB/sec)
2017-06-13T00:35:23.807 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-13T00:35:23.815 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 428.3 KB/sec)
2017-06-13T00:35:23.819 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-13T00:35:23.829 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 100.3 KB/sec)
2017-06-13T00:35:23.834 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-13T00:35:23.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 212.0 KB/sec)
2017-06-13T00:35:23.847 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-13T00:35:23.854 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 252.0 KB/sec)
2017-06-13T00:35:23.859 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-13T00:35:23.870 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 241.3 KB/sec)
2017-06-13T00:35:23.875 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-13T00:35:23.885 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1761.6 KB/sec)
2017-06-13T00:35:23.889 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-13T00:35:23.898 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 1047.5 KB/sec)
2017-06-13T00:35:23.901 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-13T00:35:23.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 494.9 KB/sec)
2017-06-13T00:35:23.914 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-13T00:35:23.922 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
 (3 KB at 288.2 KB/sec)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #78

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] [BEAM-2423] Abstract StateInternalsTest for the different 
state

[altay] Reverse removal of NativeWrite evaluator in Python DirectRunner

[tgroh] Cleanup Combine Tests with Context

--
[...truncated 1.56 MB...]
  [javadoc]   ^
  [javadoc]   symbol:   class CompletableFuture
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:66:
 error: cannot find symbol
  [javadoc]   Supplier processBundleInstructionIdSupplier,
  [javadoc]   ^
  [javadoc]   symbol:   class Supplier
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:51:
 warning - Tag @link: reference not found: DoFn.FinishBundle @FinishBundle
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:51:
 warning - Tag @link: reference not found: DoFn.FinishBundle @FinishBundle
  [javadoc] 
:34:
 warning - Tag @link: reference not found: Source.Reader
  [javadoc] 
:292:
 warning - Tag @link: reference not found: UnboundedSource.CheckpointMark
  [javadoc] 
:101:
 warning - Tag @link: reference not found: UnboundedSource.CheckpointMark
  [javadoc] 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #2342

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2436) table is not regeisted in BeamSql.query

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2436:
-
Labels: dsl_sql_merge  (was: )

> table is not regeisted in BeamSql.query
> ---
>
> Key: BEAM-2436
> URL: https://issues.apache.org/jira/browse/BEAM-2436
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2340

[Apache Jenkins Server]

See 

[1/2] beam git commit: This closes #3345

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 86e04893a -> ce3dd4583


This closes #3345


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ce3dd458
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ce3dd458
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ce3dd458

Branch: refs/heads/master
Commit: ce3dd4583294fcbb65121f5fecfc49a8b85634ad
Parents: 86e0489 f0f98c7
Author: Thomas Groh 
Authored: Mon Jun 12 15:48:38 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jun 12 15:48:38 2017 -0700

--
 .../apache/beam/sdk/transforms/CombineTest.java | 225 +--
 1 file changed, 154 insertions(+), 71 deletions(-)
--

[GitHub] beam pull request #3345: Cleanups to CombineTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3345


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3346

[altay]

This closes #3346


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/86e04893
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/86e04893
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/86e04893

Branch: refs/heads/master
Commit: 86e04893a14d9b6aa29b78e25dc064879b712fb3
Parents: f9d51aa 809f178
Author: Ahmet Altay 
Authored: Mon Jun 12 15:43:44 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Jun 12 15:43:44 2017 -0700

--
 .../runners/direct/transform_evaluator.py   | 62 +++-
 1 file changed, 61 insertions(+), 1 deletion(-)
--

[GitHub] beam pull request #3346: Reverse removal of NativeWrite evaluator in Python ...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3346

Reverse removal of NativeWrite evaluator in Python DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam fix-nativewrite

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3346


commit 5a5c8208a0e0e4b55e8c5fe8ba3a6228e6c5856a
Author: Charles Chen 
Date:   2017-06-12T21:17:50Z

Reverse removal of NativeWrite evaluator in Python DirectRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3313: [BEAM-2423] Abstract StateInternalsTest for the dif...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3313


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2423] Abstract StateInternalsTest for the different state internals

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master 1597f3ca6 -> f9d51aa5c


[BEAM-2423] Abstract StateInternalsTest for the different state internals


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8362bdb9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8362bdb9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8362bdb9

Branch: refs/heads/master
Commit: 8362bdb9cd35cc02ed179b3a64fd72f1264a99be
Parents: 1597f3c
Author: JingsongLi 
Authored: Thu Jun 8 01:31:34 2017 +0800
Committer: Aljoscha Krettek 
Committed: Mon Jun 12 11:51:29 2017 +0200

--
 pom.xml |   7 +
 .../core/InMemoryStateInternalsTest.java| 555 ++
 .../beam/runners/core/StateInternalsTest.java   | 573 +++
 runners/flink/pom.xml   |   8 +
 .../streaming/FlinkStateInternalsTest.java  | 348 +--
 5 files changed, 641 insertions(+), 850 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8362bdb9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 805a8d6..9373a40 100644
--- a/pom.xml
+++ b/pom.xml
@@ -511,6 +511,13 @@
 
   
 org.apache.beam
+beam-runners-core-java
+${project.version}
+test-jar
+  
+
+  
+org.apache.beam
 beam-runners-direct-java
 ${project.version}
   

http://git-wip-us.apache.org/repos/asf/beam/blob/8362bdb9/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryStateInternalsTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryStateInternalsTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryStateInternalsTest.java
index b526305..335c2f8 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryStateInternalsTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryStateInternalsTest.java
@@ -17,545 +17,58 @@
  */
 package org.apache.beam.runners.core;
 
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasItems;
-import static org.hamcrest.Matchers.not;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
 
-import java.util.Arrays;
-import java.util.Map;
-import java.util.Objects;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.state.BagState;
-import org.apache.beam.sdk.state.CombiningState;
-import org.apache.beam.sdk.state.GroupingState;
-import org.apache.beam.sdk.state.MapState;
-import org.apache.beam.sdk.state.ReadableState;
-import org.apache.beam.sdk.state.SetState;
-import org.apache.beam.sdk.state.ValueState;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
+import org.apache.beam.sdk.state.State;
 import org.hamcrest.Matchers;
-import org.joda.time.Instant;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
+import org.junit.runners.Suite;
 
 /**
- * Tests for {@link InMemoryStateInternals}.
+ * Tests for {@link InMemoryStateInternals}. This is based on {@link 
StateInternalsTest}.
  */
-@RunWith(JUnit4.class)
+@RunWith(Suite.class)
+@Suite.SuiteClasses({
+InMemoryStateInternalsTest.StandardStateInternalsTests.class,
+InMemoryStateInternalsTest.OtherTests.class
+})
 public class InMemoryStateInternalsTest {
-  private static final BoundedWindow WINDOW_1 = new IntervalWindow(new 
Instant(0), new Instant(10));
-  private static final StateNamespace NAMESPACE_1 = new 
StateNamespaceForTest("ns1");
-  private static final StateNamespace NAMESPACE_2 = new 
StateNamespaceForTest("ns2");
-  private static final StateNamespace NAMESPACE_3 = new 
StateNamespaceForTest("ns3");
 
-  private static final StateTag STRING_VALUE_ADDR =
-  StateTags.value("stringValue", StringUtf8Coder.of());
-  private static final StateTag>
-  SUM_INTEGER_ADDR = StateTags.combiningValueFromInputInternal(
-  "sumInteger", VarIntCoder.of(), Sum.ofIntegers());
-  private static final StateTag STRING_BAG_ADDR =

[GitHub] beam pull request #3344: Improves message when transitively serializing Pipe...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3344

Improves message when transitively serializing PipelineOptions

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam options-serializable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3344.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3344


commit 32de194c3c337bdec6d7dd432e7ab9d363df140d
Author: Eugene Kirpichov 
Date:   2017-06-12T18:51:39Z

Improves message when transitively serializing PipelineOptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #125

[Apache Jenkins Server]

See 


--
[...truncated 2.64 MB...]
2017-06-12T18:35:29.015 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 363.1 KB/sec)
2017-06-12T18:35:29.015 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-12T18:35:29.046 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 414.0 KB/sec)
2017-06-12T18:35:29.046 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-12T18:35:29.078 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 67.9 KB/sec)
2017-06-12T18:35:29.078 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-12T18:35:29.109 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 50.7 KB/sec)
2017-06-12T18:35:29.109 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-12T18:35:29.140 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 169.4 KB/sec)
2017-06-12T18:35:29.140 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-12T18:35:29.172 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 78.0 KB/sec)
2017-06-12T18:35:29.172 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-12T18:35:29.203 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 110.5 KB/sec)
2017-06-12T18:35:29.203 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-12T18:35:29.234 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 32.4 KB/sec)
2017-06-12T18:35:29.234 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-12T18:35:29.265 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 54.7 KB/sec)
2017-06-12T18:35:29.265 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-12T18:35:29.297 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 55.1 KB/sec)
2017-06-12T18:35:29.297 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-12T18:35:29.328 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 85.6 KB/sec)
2017-06-12T18:35:29.328 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-12T18:35:29.359 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 568.3 KB/sec)
2017-06-12T18:35:29.359 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-12T18:35:29.390 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 304.1 KB/sec)
2017-06-12T18:35:29.390 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-12T18:35:29.422 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 139.2 KB/sec)
2017-06-12T18:35:29.422 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-12T18:35:29.453 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
 (3 KB at 74.4 KB/sec)
2017-06-12T18:35:29.453 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
2017-06-12T18:35:29.484 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
 (4 KB at 113.4 KB/sec)
2017-06-12T18:35:29.484 [INFO] Downloading: 

[jira] [Resolved] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-2405.
-
   Resolution: Fixed
Fix Version/s: 2.1.0

> Create a BigQuery sink for streaming using PTransform
> -
>
> Key: BEAM-2405
> URL: https://issues.apache.org/jira/browse/BEAM-2405
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2338

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #77

[Apache Jenkins Server]

See 


--
[...truncated 1016.28 KB...]
2017-06-12\T\18:12:20.864 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding org.json:json:jar:20160810 from the 
shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding com.google.api:gax:jar:1.1.0 from 
the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding org.threeten:threetenbp:jar:1.3.3 
from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:proto-google-iam-v1:jar:0.1.11 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.cloud:google-cloud-spanner:jar:0.16.0-beta from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.cloud:google-cloud-core-grpc:jar:1.0.0-rc3 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding com.google.api:gax-grpc:jar:0.15.0 
from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.9 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.9 from 
the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.9 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.9 from 
the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.9 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.9 from the shaded jar.
2017-06-12\T\18:12:20.864 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Including 
org.apache.beam:beam-runners-core-construction-java:jar:2.1.0-SNAPSHOT in the 
shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.http-client:google-http-client:jar:1.22.0 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding commons-codec:commons-codec:jar:1.3 
from the shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.http-client:google-http-client-jackson2:jar:1.22.0 from the shaded 
jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev196-1.22.0 from the 
shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-06-12\T\18:12:20.865 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.1 from the shaded jar.
2017-06-12\T\18:12:20.865 

[jira] [Commented] (BEAM-2433) unify SQL/Sql words in class name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046827#comment-16046827
 ] 

ASF GitHub Bot commented on BEAM-2433:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3340


> unify SQL/Sql words in class name
> -
>
> Key: BEAM-2433
> URL: https://issues.apache.org/jira/browse/BEAM-2433
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>
> Use {{Sql}} for class name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3340: [BEAM-2433] unify SQL/Sql words in class name

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3340


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046824#comment-16046824
 ] 

Stephen Sisk commented on BEAM-1323:


can you elaborate on what exactly the SplittingFn would do? What parameters 
does it take, what does it return, how is that used by jdbcio and what are 
databases where we think it would work well?

> Add parallelism/splitting in JdbcIO
> ---
>
> Key: BEAM-1323
> URL: https://issues.apache.org/jira/browse/BEAM-1323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Now, the JDBC IO is basically a {{DoFn}} executed with a {{ParDo}}. So, it 
> means that parallelism is "limited" and executed on one executor.
> We can imagine to create several JDBC {{BoundedSource}}s splitting the SQL 
> query in  subset (for instance using row id paging or any "splitting/limit" 
> we can figure based on the original SQL query) (something similar to what 
> Sqoop is doing).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump #174

[Apache Jenkins Server]

See 


Changes:

[mingmxu] update JavaDoc for BoundedWindow

[klk] Restore non-recursive globs in GcsUtil

[dhalperi] [BEAM-1702] Document how to use BigtableIO with local emulator

[klk] [BEAM-659] WindowFn#isCompatible should provide a meaningful reason

[kirpichov] Remove incorrect `Message` bound on writeAvros

[tgroh] Add ParDos

[kirpichov] Splits SplittableParDo into a core-construction part and a 
KWI-related

[dhalperi] [BEAM-2318] Ensure that tables that don't exist are not created

[iemejia] [BEAM-2391] Clone Scan in HBaseReader

[lcwik] [BEAM-2252] Use pluginManagement instead of redefining

[altay] update dataflow python container

[klk] [BEAM-2164] Fix generate-sources.sh if directory contains space

[klk] Move snappy-java version to dependencyManagement

[kirpichov] Comply with byte limit for Datastore Commit.

[kirpichov] Make the large entities test neater.

[kirpichov] End-to-end test for large entity writes.

[lcwik] [BEAM-1345] Remove FileSystems.setDefaultConfigInWorkers since Dataflow

[lcwik] [BEAM-2293] Rename Top.{Largest,Smallest=>Natural,Reversed}

[lcwik] [BEAM-2253] Use resource transformer plugin to use versions defined in

[lcwik] [BEAM-2334] Fix OutOfMemoryError

[klk] Add TransformPayloadTranslatorRegistrar

[klk] Remove Pipeline reference from TransformHierarchy

[tgroh] Fix GroupAlsoByWindowEvaluatorFactory doc

[klk] Minor touch-ups to new BoundedWindow javadoc

[tgroh] Remove Unused DoFnInfo methods

[tgroh] Extract the Main Input PCollection in ParDos

[tgroh] Cleanup k8s scripts naming & don't create insecure svc by default

[altay] [BEAM-1999] Remove resolved todo from value providers

[altay] Update Dataflow BATCH_ENVIRONMENT_MAJOR_VERSION to 6

[tgroh] Add ReadTranslator

[tgroh] Include Additional PTransform inputs in Transform Nodes

[klk] Add WindowIntoTranslator

[klk] Fix compile error from bad merge

[altay] Comply with byte limit for Datastore Commit.

[dhalperi] Re-rename fat jar so that install doesn't install the bundled jar as 
the

[tgroh] Revert "Include Additional PTransform inputs in Transform Nodes"

[robertwb] Automatically convert examples to use with syntax.

[klk] Move StepContext to top level

[klk] Remove StepContext.noteOutput

[klk] Rename BaseExecutionContext.StepContext to BaseStepContext

[klk] Move BaseStepContext to the top level

[klk] Remove extraneous ExecutionContext parameter to BaseStepContext

[klk] Implement StepContext directly in the DirectRunner

[klk] Remove writePCollectionViewData from the Beam codebase

[klk] Inline and delete BaseExecutionContext

[klk] Delete unused ExecutionContext

[klk] Remove unused StepContext name methods

[klk] Delete unused remnants in DirectExecutionContext

[klk] Delete unused BaseStepContext

[klk] Shorten excessive name in DirectExecutionContext

[klk] Revise StepContext javadoc

[klk] Remove unused pieces of DirectStepContext

[altay] Fix lint error in datastoreio_test

[chamikara] [BEAM-2338] Fix the limit counter in gcsio reads

[klk] Rename PCollections to PCollectionTranslation

[klk] Rename PTransforms to PTransformTranslation

[klk] Rename ParDos to ParDoTranslation

[klk] Rename WindowIntoTranslator to WindowIntoTranslation

[klk] Rename ReadTranslator to ReadTranslation

[klk] Rename Coders to CoderTranslation

[klk] Rename WindowingStrategies to WindowingStrategyTranslation

[klk] Rename Triggers to TriggerTranslation

[altay] upgrading python sdk dependencies

[klk] Do not ever shrink allowed lateness

[klk] Adds large key tests to GroupByKeyTest

[tgroh] Add CombineTranslation

[dhalperi] Initial implementation of SpannerIO.Write

[dhalperi] Minor style, compilation, javadoc fixups

[dhalperi] Fix spanner dependency management

[dhalperi] Delete SpannerCSVLoader

[dhalperi] Refine Spanner API tests

[dhalperi] SpannerIO.Write cleanup and style fixes

[robertwb] Automatically generate Python proto and grpc files.

[robertwb] Remove auto-generated proto and grpc files.

[robertwb] A couple of worker fixes.

[robertwb] Adding a snippet for metrics

[iemejia] Update maven-dependency-plugin to version 3.0.1

[klk] Make SdkComponents public for TransformPayloadTranslator

[klk] Centralize primitive URNs in PTransformTranslation class

[klk] Add URN for Splittable ProcessElement pseudo-primitive

[klk] Allow getting URN for class of transform via translator

[klk] Add registration for Read and WindowInto translators

[klk] Register ReadTranslator

[klk] Register WindowIntoTranslator

[klk] Add trivial FlattenTranslator to access URN

[klk] Add transform-analysis helpers to ReadTranslation

[klk] Add RawPTransform, which can just vend its URN and payload

[klk] URNs for DirectRunner TransformEvaluator and RootInputProvider

[robertwb] More robust gen_protos on jenkins.

[tgroh] Clarify that PTransform#expand shouldn't be called

[klk] Fixup CombineTranslation

[kirpichov] 

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046784#comment-16046784
 ] 

Jean-Baptiste Onofré commented on BEAM-1323:


The idea is to allow user provide a Splitting Fn, adapted to the targeted 
database. I don't think it's possible to implement a generic way of doing 
splitting in JDBC, the idea is more to delegate. Thoughts ?

> Add parallelism/splitting in JdbcIO
> ---
>
> Key: BEAM-1323
> URL: https://issues.apache.org/jira/browse/BEAM-1323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Now, the JDBC IO is basically a {{DoFn}} executed with a {{ParDo}}. So, it 
> means that parallelism is "limited" and executed on one executor.
> We can imagine to create several JDBC {{BoundedSource}}s splitting the SQL 
> query in  subset (for instance using row id paging or any "splitting/limit" 
> we can figure based on the original SQL query) (something similar to what 
> Sqoop is doing).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046775#comment-16046775
 ] 

Stephen Sisk commented on BEAM-1323:


+1 to eugene's comment. I don't believe we can reliably do this in jdbc IO in a 
way that works across  databases - thus, any implementation of this proposal 
would likely result in bad experiences for many users.

Unless you have a concrete proposal that we can prove works across many 
databases, I think we should close this issue.

> Add parallelism/splitting in JdbcIO
> ---
>
> Key: BEAM-1323
> URL: https://issues.apache.org/jira/browse/BEAM-1323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Now, the JDBC IO is basically a {{DoFn}} executed with a {{ParDo}}. So, it 
> means that parallelism is "limited" and executed on one executor.
> We can imagine to create several JDBC {{BoundedSource}}s splitting the SQL 
> query in  subset (for instance using row id paging or any "splitting/limit" 
> we can figure based on the original SQL query) (something similar to what 
> Sqoop is doing).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[34/50] [abbrv] beam git commit: [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and ES server versions

[kenn]

[BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and ES 
server versions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7caea7a8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7caea7a8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7caea7a8

Branch: refs/heads/gearpump-runner
Commit: 7caea7a845eff072a647baf69b9b004db4523652
Parents: e980ae9
Author: Etienne Chauchot 
Authored: Mon Jun 5 16:21:58 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Fri Jun 9 07:31:06 2017 +0200

--
 .../sdk/io/common/IOTestPipelineOptions.java|  6 +-
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |  4 +-
 .../elasticsearch/ElasticSearchIOTestUtils.java | 81 +++-
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java | 14 ++--
 .../io/elasticsearch/ElasticsearchIOTest.java   | 36 +
 .../elasticsearch/ElasticsearchTestDataSet.java | 37 -
 6 files changed, 87 insertions(+), 91 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
--
diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index 387fd22..25ab929 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -71,11 +71,7 @@ public interface IOTestPipelineOptions extends 
TestPipelineOptions {
   Integer getElasticsearchHttpPort();
   void setElasticsearchHttpPort(Integer value);
 
-  @Description("Tcp port for elasticsearch server")
-  @Default.Integer(9300)
-  Integer getElasticsearchTcpPort();
-  void setElasticsearchTcpPort(Integer value);
-
+  /* Cassandra */
   @Description("Host for Cassandra server (host name/ip address)")
   @Default.String("cassandra-host")
   String getCassandraHost();

http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
--
diff --git 
a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
 
b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
index f6ceef2..e3965dc 100644
--- 
a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
+++ 
b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
@@ -139,7 +139,7 @@ public class ElasticsearchIO {
 
   private static final ObjectMapper mapper = new ObjectMapper();
 
-  private static JsonNode parseResponse(Response response) throws IOException {
+  static JsonNode parseResponse(Response response) throws IOException {
 return mapper.readValue(response.getEntity().getContent(), JsonNode.class);
   }
 
@@ -264,7 +264,7 @@ public class ElasticsearchIO {
   builder.addIfNotNull(DisplayData.item("username", getUsername()));
 }
 
-private RestClient createClient() throws MalformedURLException {
+RestClient createClient() throws MalformedURLException {
   HttpHost[] hosts = new HttpHost[getAddresses().size()];
   int i = 0;
   for (String address : getAddresses()) {

http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
--
diff --git 
a/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
 
b/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
index b0d161f..203963d 100644
--- 
a/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
+++ 
b/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
@@ -17,19 +17,17 @@
  */
 package org.apache.beam.sdk.io.elasticsearch;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
-import 
org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequest;
-import 
org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse;
-import 

[14/50] [abbrv] beam git commit: This closes #3303

[kenn]

This closes #3303


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e066a9d6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e066a9d6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e066a9d6

Branch: refs/heads/gearpump-runner
Commit: e066a9d6dfc4d76f2d7d49de9f744b978982bb81
Parents: 86e1fab ccf7344
Author: Thomas Groh 
Authored: Thu Jun 8 09:13:57 2017 -0700
Committer: Thomas Groh 
Committed: Thu Jun 8 09:13:57 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  59 +--
 .../translation/ApexPipelineTranslator.java |  16 +-
 .../construction/RunnerPCollectionView.java |   8 +
 .../apache/beam/runners/direct/DirectGraph.java |   4 +
 .../beam/runners/direct/DirectGraphVisitor.java |  22 +-
 .../runners/direct/ViewEvaluatorFactory.java|   8 +-
 .../runners/direct/ViewOverrideFactory.java |  29 +-
 .../beam/runners/direct/DirectGraphs.java   |   7 +
 .../runners/direct/EvaluationContextTest.java   |   5 +-
 .../ImmutabilityEnforcementFactoryTest.java |   4 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 +
 .../runners/direct/TransformExecutorTest.java   |   1 +
 .../direct/ViewEvaluatorFactoryTest.java|   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |  16 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   1 +
 .../runners/direct/WatermarkManagerTest.java|   1 +
 .../runners/flink/CreateStreamingFlinkView.java | 154 
 .../flink/FlinkStreamingPipelineTranslator.java |  36 +-
 .../FlinkStreamingTransformTranslators.java |   8 +-
 .../flink/FlinkStreamingViewOverrides.java  | 372 ---
 .../runners/dataflow/BatchViewOverrides.java| 182 +++--
 .../runners/dataflow/CreateDataflowView.java|   8 +-
 .../dataflow/DataflowPipelineTranslator.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 +-
 .../dataflow/StreamingViewOverrides.java|  10 +-
 .../DataflowPipelineTranslatorTest.java |   6 +-
 .../spark/translation/TransformTranslator.java  |  50 +--
 .../beam/sdk/runners/TransformHierarchy.java|  46 ++-
 .../org/apache/beam/sdk/transforms/Combine.java |  17 +-
 .../org/apache/beam/sdk/transforms/View.java|  38 +-
 .../org/apache/beam/sdk/values/PCollection.java |  12 +
 .../beam/sdk/values/PCollectionViews.java   |  14 +
 .../org/apache/beam/sdk/values/PValueBase.java  |  12 -
 .../sdk/testing/PCollectionViewTesting.java |   8 +
 34 files changed, 458 insertions(+), 730 deletions(-)
--

[41/50] [abbrv] beam git commit: Visit a Transform Hierarchy in Topological Order

[kenn]

Visit a Transform Hierarchy in Topological Order

This reverts commit 6ad6433ec0c02aec8656e9e3b27f6e0f974f8ece.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/261e7df2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/261e7df2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/261e7df2

Branch: refs/heads/gearpump-runner
Commit: 261e7df2b860fe82d9f401e2621b020fe2020fea
Parents: d2c4093
Author: Thomas Groh 
Authored: Tue Jun 6 16:15:19 2017 -0700
Committer: Thomas Groh 
Committed: Fri Jun 9 14:44:42 2017 -0700

--
 .../spark/translation/StorageLevelTest.java |   4 +-
 .../beam/sdk/runners/TransformHierarchy.java|  79 +++-
 .../sdk/runners/TransformHierarchyTest.java | 197 +++
 3 files changed, 274 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/261e7df2/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/StorageLevelTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/StorageLevelTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/StorageLevelTest.java
index 8f2e681..8bd6dae 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/StorageLevelTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/StorageLevelTest.java
@@ -58,12 +58,12 @@ public class StorageLevelTest {
 
   @Test
   public void test() throws Exception {
-PCollection pCollection = pipeline.apply(Create.of("foo"));
+PCollection pCollection = pipeline.apply("CreateFoo", 
Create.of("foo"));
 
 // by default, the Spark runner doesn't cache the RDD if it accessed only 
one time.
 // So, to "force" the caching of the RDD, we have to call the RDD at least 
two time.
 // That's why we are using Count fn on the PCollection.
-pCollection.apply(Count.globally());
+pCollection.apply("CountAll", Count.globally());
 
 PCollection output = pCollection.apply(new 
StorageLevelPTransform());
 

http://git-wip-us.apache.org/repos/asf/beam/blob/261e7df2/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
index ee1ce7b..5e048eb 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
@@ -208,7 +208,7 @@ public class TransformHierarchy {
   public Set visit(PipelineVisitor visitor) {
 finishSpecifying();
 Set visitedValues = new HashSet<>();
-root.visit(visitor, visitedValues);
+root.visit(visitor, visitedValues, new HashSet(), new 
HashSet());
 return visitedValues;
   }
 
@@ -503,10 +503,60 @@ public class TransformHierarchy {
 /**
  * Visit the transform node.
  *
+ * The visit proceeds in the following order:
+ *
+ * 
+ *   Visit all input {@link PValue PValues} returned by the flattened 
expansion of {@link
+ *   Node#getInputs()}.
+ *   If the node is a composite:
+ *   
+ * Enter the node via {@link 
PipelineVisitor#enterCompositeTransform(Node)}.
+ * If the result of {@link 
PipelineVisitor#enterCompositeTransform(Node)} was {@link
+ * CompositeBehavior#ENTER_TRANSFORM}, visit each child node 
of this {@link Node}.
+ * Leave the node via {@link 
PipelineVisitor#leaveCompositeTransform(Node)}.
+ *   
+ *   If the node is a primitive, visit it via {@link
+ *   PipelineVisitor#visitPrimitiveTransform(Node)}.
+ *   Visit each {@link PValue} that was output by this node.
+ * 
+ *
+ * Additionally, the following ordering restrictions are observed:
+ *
+ * 
+ *   A {@link Node} will be visited after its enclosing node has been 
entered and before its
+ *   enclosing node has been left
+ *   A {@link Node} will not be visited if any enclosing {@link Node} 
has returned {@link
+ *   CompositeBehavior#DO_NOT_ENTER_TRANSFORM} from the call to {@link
+ *   PipelineVisitor#enterCompositeTransform(Node)}.
+ *   A {@link PValue} will only be visited after the {@link Node} that 
originally produced
+ *   it has been visited.
+ * 
+ *
  * Provides an ordered visit of the input values, the primitive 
transform (or child nodes for
  * composite transforms), then the output values.
  */
-  

[16/50] [abbrv] beam git commit: Expand all PValues to component PCollections always

[kenn]

Expand all PValues to component PCollections always

Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ccf73448
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ccf73448
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ccf73448

Branch: refs/heads/gearpump-runner
Commit: ccf7344820d6c69ca922aa3176dc141718382629
Parents: 86e1fab
Author: Thomas Groh 
Authored: Thu Jun 1 18:39:58 2017 -0700
Committer: Thomas Groh 
Committed: Thu Jun 8 09:13:57 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  59 +--
 .../translation/ApexPipelineTranslator.java |  16 +-
 .../construction/RunnerPCollectionView.java |   8 +
 .../apache/beam/runners/direct/DirectGraph.java |   4 +
 .../beam/runners/direct/DirectGraphVisitor.java |  22 +-
 .../runners/direct/ViewEvaluatorFactory.java|   8 +-
 .../runners/direct/ViewOverrideFactory.java |  29 +-
 .../beam/runners/direct/DirectGraphs.java   |   7 +
 .../runners/direct/EvaluationContextTest.java   |   5 +-
 .../ImmutabilityEnforcementFactoryTest.java |   4 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 +
 .../runners/direct/TransformExecutorTest.java   |   1 +
 .../direct/ViewEvaluatorFactoryTest.java|   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |  16 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   1 +
 .../runners/direct/WatermarkManagerTest.java|   1 +
 .../runners/flink/CreateStreamingFlinkView.java | 154 
 .../flink/FlinkStreamingPipelineTranslator.java |  36 +-
 .../FlinkStreamingTransformTranslators.java |   8 +-
 .../flink/FlinkStreamingViewOverrides.java  | 372 ---
 .../runners/dataflow/BatchViewOverrides.java| 182 +++--
 .../runners/dataflow/CreateDataflowView.java|   8 +-
 .../dataflow/DataflowPipelineTranslator.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 +-
 .../dataflow/StreamingViewOverrides.java|  10 +-
 .../DataflowPipelineTranslatorTest.java |   6 +-
 .../spark/translation/TransformTranslator.java  |  50 +--
 .../beam/sdk/runners/TransformHierarchy.java|  46 ++-
 .../org/apache/beam/sdk/transforms/Combine.java |  17 +-
 .../org/apache/beam/sdk/transforms/View.java|  38 +-
 .../org/apache/beam/sdk/values/PCollection.java |  12 +
 .../beam/sdk/values/PCollectionViews.java   |  14 +
 .../org/apache/beam/sdk/values/PValueBase.java  |  12 -
 .../sdk/testing/PCollectionViewTesting.java |   8 +
 34 files changed, 458 insertions(+), 730 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index c595b3f..95b354a 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -62,8 +62,6 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
-import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.View.AsIterable;
 import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionTuple;
@@ -214,7 +212,7 @@ public class ApexRunner extends 
PipelineRunner {
* @param  The type associated with the {@link PCollectionView} used 
as a side input
*/
   public static class CreateApexPCollectionView
-  extends PTransform, PCollectionView> {
+  extends PTransform {
 private static final long serialVersionUID = 1L;
 private PCollectionView view;
 
@@ -228,7 +226,13 @@ public class ApexRunner extends 
PipelineRunner {
 }
 
 @Override
-public PCollectionView expand(PCollection input) {
+public PCollection expand(PCollection input) {
+  return PCollection.createPrimitiveOutputInternal(
+  

[05/50] [abbrv] beam git commit: [BEAM-2421] Swap to use an Impulse primitive + DoFn for Create when executing with the Fn API.

[kenn]

[BEAM-2421] Swap to use an Impulse primitive + DoFn for Create when executing 
with the Fn API.

This closes #3312


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/caecac3b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/caecac3b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/caecac3b

Branch: refs/heads/gearpump-runner
Commit: caecac3b4acb5bfa6e36143d3868b2d80ab119da
Parents: 609016d 1cdb80c
Author: Luke Cwik 
Authored: Wed Jun 7 13:43:38 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jun 7 13:43:38 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 154 ++-
 1 file changed, 146 insertions(+), 8 deletions(-)
--

[19/50] [abbrv] beam git commit: Refine Python DirectRunner watermark advancement behavior

[kenn]

Refine Python DirectRunner watermark advancement behavior

This change helps prepare for streaming pipeline execution.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3e049020
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3e049020
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3e049020

Branch: refs/heads/gearpump-runner
Commit: 3e04902008b410269b23179dc2146623ff1fbd0a
Parents: d81ed21
Author: Charles Chen 
Authored: Wed Jun 7 17:46:36 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:55:44 2017 -0700

--
 .../runners/direct/watermark_manager.py | 20 +---
 sdks/python/apache_beam/utils/timestamp.py  |  5 +
 2 files changed, 22 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3e049020/sdks/python/apache_beam/runners/direct/watermark_manager.py
--
diff --git a/sdks/python/apache_beam/runners/direct/watermark_manager.py 
b/sdks/python/apache_beam/runners/direct/watermark_manager.py
index 3a13539..0d7cd4f 100644
--- a/sdks/python/apache_beam/runners/direct/watermark_manager.py
+++ b/sdks/python/apache_beam/runners/direct/watermark_manager.py
@@ -25,6 +25,7 @@ from apache_beam import pipeline
 from apache_beam import pvalue
 from apache_beam.utils.timestamp import MAX_TIMESTAMP
 from apache_beam.utils.timestamp import MIN_TIMESTAMP
+from apache_beam.utils.timestamp import TIME_GRANULARITY
 
 
 class WatermarkManager(object):
@@ -193,9 +194,22 @@ class _TransformWatermarks(object):
 
   def refresh(self):
 with self._lock:
-  pending_holder = (WatermarkManager.WATERMARK_NEG_INF
-if self._pending else
-WatermarkManager.WATERMARK_POS_INF)
+  min_pending_timestamp = WatermarkManager.WATERMARK_POS_INF
+  has_pending_elements = False
+  for input_bundle in self._pending:
+# TODO(ccy): we can have the Bundle class keep track of the minimum
+# timestamp so we don't have to do an iteration here.
+for wv in input_bundle.get_elements_iterable():
+  has_pending_elements = True
+  if wv.timestamp < min_pending_timestamp:
+min_pending_timestamp = wv.timestamp
+
+  # If there is a pending element with a certain timestamp, we can at most
+  # advance our watermark to the maximum timestamp less than that
+  # timestamp.
+  pending_holder = WatermarkManager.WATERMARK_POS_INF
+  if has_pending_elements:
+pending_holder = min_pending_timestamp - TIME_GRANULARITY
 
   input_watermarks = [
   tw.output_watermark for tw in self._input_transform_watermarks]

http://git-wip-us.apache.org/repos/asf/beam/blob/3e049020/sdks/python/apache_beam/utils/timestamp.py
--
diff --git a/sdks/python/apache_beam/utils/timestamp.py 
b/sdks/python/apache_beam/utils/timestamp.py
index 5d1b48c..b3e840e 100644
--- a/sdks/python/apache_beam/utils/timestamp.py
+++ b/sdks/python/apache_beam/utils/timestamp.py
@@ -208,3 +208,8 @@ class Duration(object):
   def __mod__(self, other):
 other = Duration.of(other)
 return Duration(micros=self.micros % other.micros)
+
+
+# The minimum granularity / interval expressible in a Timestamp / Duration
+# object.
+TIME_GRANULARITY = Duration(micros=1)

[27/50] [abbrv] beam git commit: Closes #3325

[kenn]

Closes #3325


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c5a70ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c5a70ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c5a70ed

Branch: refs/heads/gearpump-runner
Commit: 7c5a70edb2d19f22b0a6b81ae516762703358986
Parents: 4aef938 36aea2d
Author: Robert Bradshaw 
Authored: Thu Jun 8 13:58:31 2017 -0700
Committer: Robert Bradshaw 
Committed: Thu Jun 8 13:58:31 2017 -0700

--
 sdks/python/apache_beam/runners/worker/operation_specs.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--

[46/50] [abbrv] beam git commit: Port DirectRunner WriteFiles override to SDK-agnostic APIs

[kenn]

Port DirectRunner WriteFiles override to SDK-agnostic APIs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed6bd18b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed6bd18b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ed6bd18b

Branch: refs/heads/gearpump-runner
Commit: ed6bd18bffe8a51d5fc2a59ff9aaa731b196d58a
Parents: 02dbaef
Author: Kenneth Knowles 
Authored: Fri May 26 16:07:45 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 19:56:52 2017 -0700

--
 .../core/construction/PTransformMatchers.java   | 17 ---
 .../direct/WriteWithShardingFactory.java| 30 ++--
 .../direct/WriteWithShardingFactoryTest.java| 26 +++--
 3 files changed, 52 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ed6bd18b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
index c339891..0d27241 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
@@ -17,13 +17,14 @@
  */
 package org.apache.beam.runners.core.construction;
 
+import static 
org.apache.beam.runners.core.construction.PTransformTranslation.WRITE_FILES_TRANSFORM_URN;
+
 import com.google.common.base.MoreObjects;
 import java.io.IOException;
 import java.util.HashSet;
 import java.util.Set;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.sdk.io.WriteFiles;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.runners.PTransformMatcher;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -359,10 +360,18 @@ public class PTransformMatchers {
 return new PTransformMatcher() {
   @Override
   public boolean matches(AppliedPTransform application) {
-if (PTransformTranslation.WRITE_FILES_TRANSFORM_URN.equals(
+if (WRITE_FILES_TRANSFORM_URN.equals(
 
PTransformTranslation.urnForTransformOrNull(application.getTransform( {
-  WriteFiles write = (WriteFiles) application.getTransform();
-  return write.getSharding() == null && write.getNumShards() == null;
+  try {
+return WriteFilesTranslation.isRunnerDeterminedSharding(
+(AppliedPTransform) application);
+  } catch (IOException exc) {
+throw new RuntimeException(
+String.format(
+"Transform with URN %s failed to parse: %s",
+WRITE_FILES_TRANSFORM_URN, application.getTransform()),
+exc);
+  }
 }
 return false;
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/ed6bd18b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
index 65a5a19..d8734a1 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
@@ -21,11 +21,13 @@ package org.apache.beam.runners.direct;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Supplier;
 import com.google.common.base.Suppliers;
+import java.io.IOException;
 import java.io.Serializable;
 import java.util.Collections;
 import java.util.Map;
 import java.util.concurrent.ThreadLocalRandom;
 import org.apache.beam.runners.core.construction.PTransformReplacements;
+import org.apache.beam.runners.core.construction.WriteFilesTranslation;
 import org.apache.beam.sdk.io.WriteFiles;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.runners.PTransformOverrideFactory;
@@ -43,23 +45,33 @@ import org.apache.beam.sdk.values.PValue;
 import org.apache.beam.sdk.values.TupleTag;
 
 /**
- * A {@link PTransformOverrideFactory} that overrides {@link WriteFiles}
- * {@link PTransform PTransforms} with an unspecified number of shards with a 
write with a
- * specified 

[30/50] [abbrv] beam git commit: Slight debuggability improvements in BigtableIO

[kenn]

Slight debuggability improvements in BigtableIO

- ByteKeyRangeTracker.splitAtPosition logs the "insane" case first.
- BigtableIO logs the split position at INFO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/82a6cb61
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/82a6cb61
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/82a6cb61

Branch: refs/heads/gearpump-runner
Commit: 82a6cb6104c0bb179832dabf4433c183743ea983
Parents: 911bfbd
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:51:15 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:51:15 2017 -0700

--
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  | 22 
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  2 +-
 2 files changed, 14 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/82a6cb61/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
index 99717a4..b889ec7 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
@@ -71,6 +71,10 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
 "Trying to return record which is before the last-returned record");
 
 if (position == null) {
+  LOG.info(
+  "Adjusting range start from {} to {} as position of first returned 
record",
+  range.getStartKey(),
+  recordStart);
   range = range.withStartKey(recordStart);
 }
 position = recordStart;
@@ -87,6 +91,15 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
 
   @Override
   public synchronized boolean trySplitAtPosition(ByteKey splitPosition) {
+// Sanity check.
+if (!range.containsKey(splitPosition)) {
+  LOG.warn(
+  "{}: Rejecting split request at {} because it is not within the 
range.",
+  this,
+  splitPosition);
+  return false;
+}
+
 // Unstarted.
 if (position == null) {
   LOG.warn(
@@ -106,15 +119,6 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
   return false;
 }
 
-// Sanity check.
-if (!range.containsKey(splitPosition)) {
-  LOG.warn(
-  "{}: Rejecting split request at {} because it is not within the 
range.",
-  this,
-  splitPosition);
-  return false;
-}
-
 range = range.withEndKey(splitPosition);
 return true;
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/82a6cb61/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 1692cda..62679bb 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -1027,7 +1027,7 @@ public class BigtableIO {
 "{}: Failed to interpolate key for fraction {}.", 
rangeTracker.getRange(), fraction, e);
 return null;
   }
-  LOG.debug(
+  LOG.info(
   "Proposing to split {} at fraction {} (key {})", rangeTracker, 
fraction, splitKey);
   BigtableSource primary;
   BigtableSource residual;

[25/50] [abbrv] beam git commit: This closes #3239: [BEAM-2371] Port DirectRunner override matchers to language-agnostic construction APIs

[kenn]

This closes #3239: [BEAM-2371] Port DirectRunner override matchers to 
language-agnostic construction APIs

  DirectRunner override matchers using Runner API
  Add Runner API oriented PTransformMatchers for DirectRunner overrides


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4aef9385
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4aef9385
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4aef9385

Branch: refs/heads/gearpump-runner
Commit: 4aef9385477f0d13bdaa345cb5a629b2fd00b867
Parents: b53e6f0 d8d9087
Author: Kenneth Knowles 
Authored: Thu Jun 8 11:37:15 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:37:15 2017 -0700

--
 .../core/construction/PTransformMatchers.java   | 94 +++-
 .../construction/PTransformTranslation.java |  7 +-
 .../construction/PTransformMatchersTest.java| 32 +++
 .../beam/runners/direct/DirectRunner.java   | 20 ++---
 4 files changed, 137 insertions(+), 16 deletions(-)
--

[11/50] [abbrv] beam git commit: [BEAM-1226] Make BytesCoder to be a known type

[kenn]

[BEAM-1226] Make BytesCoder to be a known type

This closes #3316


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a0a1bc7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a0a1bc7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a0a1bc7

Branch: refs/heads/gearpump-runner
Commit: 0a0a1bc7493ad71f1a70b1c9921ba09ecfe4c2e8
Parents: 66460cb d94ac58
Author: Luke Cwik 
Authored: Wed Jun 7 20:08:49 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jun 7 20:08:49 2017 -0700

--
 sdks/python/apache_beam/coders/coders.py  | 5 +
 sdks/python/apache_beam/runners/worker/operation_specs.py | 4 
 2 files changed, 9 insertions(+)
--

[21/50] [abbrv] beam git commit: Raise entity limit per RPC to 9MB.

[kenn]

Raise entity limit per RPC to 9MB.

This is closer to the API limit, while still leaving room for overhead. Brings
the Java SDK into line with the Python SDK.

Switch the unit test to use the size of each entity, which is what the
connector is actually using, rather than the property size (which is slightly
smaller and would cause the test to fail for some values).


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/156f326a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/156f326a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/156f326a

Branch: refs/heads/gearpump-runner
Commit: 156f326a16e15b4e22a189a2a263d11d7b273656
Parents: fdfc70e
Author: Colin Phipps 
Authored: Mon Jun 5 12:12:49 2017 +
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:57:09 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java  | 16 +---
 2 files changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/156f326a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index b198a6f..06b9c8a 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -213,7 +213,7 @@ public class DatastoreV1 {
* the mutations themselves and not the CommitRequest wrapper around them.
*/
   @VisibleForTesting
-  static final int DATASTORE_BATCH_UPDATE_BYTES_LIMIT = 5_000_000;
+  static final int DATASTORE_BATCH_UPDATE_BYTES_LIMIT = 9_000_000;
 
   /**
* Returns an empty {@link DatastoreV1.Read} builder. Configure the source 
{@code projectId},

http://git-wip-us.apache.org/repos/asf/beam/blob/156f326a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index 460049e..229b1fb 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -651,12 +651,14 @@ public class DatastoreV1Test {
   @Test
   public void testDatatoreWriterFnWithLargeEntities() throws Exception {
 List mutations = new ArrayList<>();
-int propertySize = 900_000;
+int entitySize = 0;
 for (int i = 0; i < 12; ++i) {
-  Entity.Builder entity = Entity.newBuilder().setKey(makeKey("key" + i, i 
+ 1));
-  entity.putProperties("long", makeValue(new String(new char[propertySize])
-).setExcludeFromIndexes(true).build());
-  mutations.add(makeUpsert(entity.build()).build());
+  Entity entity = Entity.newBuilder().setKey(makeKey("key" + i, i + 1))
+.putProperties("long", makeValue(new String(new char[900_000])
+  ).setExcludeFromIndexes(true).build())
+.build();
+  entitySize = entity.getSerializedSize(); // Take the size of any one 
entity.
+  mutations.add(makeUpsert(entity).build());
 }
 
 DatastoreWriterFn datastoreWriter = new 
DatastoreWriterFn(StaticValueProvider.of(PROJECT_ID),
@@ -667,10 +669,10 @@ public class DatastoreV1Test {
 
 // This test is over-specific currently; it requires that we split the 12 
entity writes into 3
 // requests, but we only need each CommitRequest to be less than 10MB in 
size.
-int propertiesPerRpc = DATASTORE_BATCH_UPDATE_BYTES_LIMIT / propertySize;
+int entitiesPerRpc = DATASTORE_BATCH_UPDATE_BYTES_LIMIT / entitySize;
 int start = 0;
 while (start < mutations.size()) {
-  int end = Math.min(mutations.size(), start + propertiesPerRpc);
+  int end = Math.min(mutations.size(), start + entitiesPerRpc);
   CommitRequest.Builder commitRequest = CommitRequest.newBuilder();
   commitRequest.setMode(CommitRequest.Mode.NON_TRANSACTIONAL);
   commitRequest.addAllMutations(mutations.subList(start, end));

[47/50] [abbrv] beam git commit: Port DirectRunner TestStream override to SDK-agnostic APIs

[kenn]

Port DirectRunner TestStream override to SDK-agnostic APIs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eaaf45fa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eaaf45fa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eaaf45fa

Branch: refs/heads/gearpump-runner
Commit: eaaf45fa33d500a9f0fd0c2861aac4889ee5086c
Parents: ed6bd18
Author: Kenneth Knowles 
Authored: Thu Jun 8 13:39:32 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 19:56:52 2017 -0700

--
 .../construction/TestStreamTranslation.java | 49 +++-
 .../direct/TestStreamEvaluatorFactory.java  | 20 ++--
 .../org/apache/beam/sdk/testing/TestStream.java | 12 +
 3 files changed, 75 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/eaaf45fa/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
index 90e6304..515de57 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
@@ -18,6 +18,9 @@
 
 package org.apache.beam.runners.core.construction;
 
+import static com.google.common.base.Preconditions.checkArgument;
+import static 
org.apache.beam.runners.core.construction.PTransformTranslation.TEST_STREAM_TRANSFORM_URN;
+
 import com.google.auto.service.AutoService;
 import com.google.protobuf.Any;
 import com.google.protobuf.ByteString;
@@ -33,6 +36,8 @@ import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.testing.TestStream;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
@@ -57,6 +62,48 @@ public class TestStreamTranslation {
 return builder.build();
   }
 
+  private static TestStream fromProto(
+  RunnerApi.TestStreamPayload testStreamPayload, RunnerApi.Components 
components)
+  throws IOException {
+
+Coder coder =
+(Coder)
+CoderTranslation.fromProto(
+components.getCodersOrThrow(testStreamPayload.getCoderId()), 
components);
+
+List events = new ArrayList<>();
+
+for (RunnerApi.TestStreamPayload.Event event : 
testStreamPayload.getEventsList()) {
+  events.add(fromProto(event, coder));
+}
+return TestStream.fromRawEvents(coder, events);
+  }
+
+  /**
+   * Converts an {@link AppliedPTransform}, which may be a rehydrated 
transform or an original
+   * {@link TestStream}, to a {@link TestStream}.
+   */
+  public static  TestStream getTestStream(
+  AppliedPTransform> application)
+  throws IOException {
+// For robustness, we don't take this shortcut:
+// if (application.getTransform() instanceof TestStream) {
+//   return application.getTransform()
+// }
+
+SdkComponents sdkComponents = SdkComponents.create();
+RunnerApi.PTransform transformProto = 
PTransformTranslation.toProto(application, sdkComponents);
+checkArgument(
+TEST_STREAM_TRANSFORM_URN.equals(transformProto.getSpec().getUrn()),
+"Attempt to get %s from a transform with wrong URN %s",
+TestStream.class.getSimpleName(),
+transformProto.getSpec().getUrn());
+RunnerApi.TestStreamPayload testStreamPayload =
+
transformProto.getSpec().getParameter().unpack(RunnerApi.TestStreamPayload.class);
+
+return (TestStream) fromProto(testStreamPayload, 
sdkComponents.toComponents());
+  }
+
   static  RunnerApi.TestStreamPayload.Event toProto(TestStream.Event 
event, Coder coder)
   throws IOException {
 switch (event.getType()) {
@@ -130,7 +177,7 @@ public class TestStreamTranslation {
   static class TestStreamTranslator implements 
TransformPayloadTranslator {
 @Override
 public String getUrn(TestStream transform) {
-  return PTransformTranslation.TEST_STREAM_TRANSFORM_URN;
+  return TEST_STREAM_TRANSFORM_URN;
 }
 
 @Override

[42/50] [abbrv] beam git commit: Roll-forward Include Additional PTransform inputs in Transform Nodes

[kenn]

Roll-forward Include Additional PTransform inputs in Transform Nodes

Update DirectGraph to have All and Non-Additional Inputs

This reverts commit 247f9bc1581984d026764b3d433cb594e700bc21


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/696f8b28
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/696f8b28
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/696f8b28

Branch: refs/heads/gearpump-runner
Commit: 696f8b28a3a17e7de81e2d46bb9774d57d6e265e
Parents: 33883ed
Author: Thomas Groh 
Authored: Tue Jun 6 17:00:09 2017 -0700
Committer: Thomas Groh 
Committed: Fri Jun 9 15:00:33 2017 -0700

--
 .../apex/translation/TranslationContext.java|   4 +-
 .../core/construction/TransformInputs.java  |  50 ++
 .../core/construction/TransformInputsTest.java  | 166 +++
 .../apache/beam/runners/direct/DirectGraph.java |  34 +++-
 .../beam/runners/direct/DirectGraphVisitor.java |  26 ++-
 .../direct/ExecutorServiceParallelExecutor.java |   2 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   9 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 +
 .../beam/runners/direct/WatermarkManager.java   |  14 +-
 .../runners/direct/DirectGraphVisitorTest.java  |  10 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../flink/FlinkBatchTranslationContext.java |   3 +-
 .../flink/FlinkStreamingTranslationContext.java |   3 +-
 .../dataflow/DataflowPipelineTranslator.java|   5 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../beam/sdk/runners/TransformHierarchy.java|  28 +++-
 18 files changed, 323 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/696f8b28/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
index aff3863..94d13e1 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
@@ -34,6 +34,7 @@ import 
org.apache.beam.runners.apex.translation.utils.ApexStateInternals;
 import 
org.apache.beam.runners.apex.translation.utils.ApexStateInternals.ApexStateBackend;
 import org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.CoderAdapterStreamCodec;
+import org.apache.beam.runners.core.construction.TransformInputs;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -93,7 +94,8 @@ class TranslationContext {
   }
 
   public  InputT getInput() {
-return (InputT) 
Iterables.getOnlyElement(getCurrentTransform().getInputs().values());
+return (InputT)
+
Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(getCurrentTransform()));
   }
 
   public Map getOutputs() {

http://git-wip-us.apache.org/repos/asf/beam/blob/696f8b28/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
new file mode 100644
index 000..2baf93a
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the 

[13/50] [abbrv] beam git commit: Add coder info to pubsub io

[kenn]

Add coder info to pubsub io


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b5852d21
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b5852d21
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b5852d21

Branch: refs/heads/gearpump-runner
Commit: b5852d212cab060321c43a5800f8585aa3649aec
Parents: 0a0a1bc
Author: Vikas Kedigehalli 
Authored: Wed Jun 7 16:28:18 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jun 7 22:55:00 2017 -0700

--
 sdks/python/apache_beam/io/gcp/pubsub.py| 32 +++-
 sdks/python/apache_beam/io/gcp/pubsub_test.py   | 28 +++--
 .../runners/dataflow/dataflow_runner.py | 23 ++
 3 files changed, 67 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b5852d21/sdks/python/apache_beam/io/gcp/pubsub.py
--
diff --git a/sdks/python/apache_beam/io/gcp/pubsub.py 
b/sdks/python/apache_beam/io/gcp/pubsub.py
index 1ba8ac0..40326e1 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub.py
@@ -40,13 +40,15 @@ __all__ = ['ReadStringsFromPubSub', 'WriteStringsToPubSub',
 class ReadStringsFromPubSub(PTransform):
   """A ``PTransform`` for reading utf-8 string payloads from Cloud Pub/Sub."""
 
-  def __init__(self, topic, subscription=None, id_label=None):
+  def __init__(self, topic=None, subscription=None, id_label=None):
 """Initializes ``ReadStringsFromPubSub``.
 
 Attributes:
-  topic: Cloud Pub/Sub topic in the form "/topics//".
-  subscription: Optional existing Cloud Pub/Sub subscription to use in the
-form "projects//subscriptions/".
+  topic: Cloud Pub/Sub topic in the form "/topics//". If
+provided then subscription must be None.
+  subscription: Existing Cloud Pub/Sub subscription to use in the
+form "projects//subscriptions/". If provided 
then
+topic must be None.
   id_label: The attribute on incoming Pub/Sub messages to use as a unique
 record identifier.  When specified, the value of this attribute (which
 can be any string that uniquely identifies the record) will be used for
@@ -55,6 +57,12 @@ class ReadStringsFromPubSub(PTransform):
 case, deduplication of the stream will be strictly best effort.
 """
 super(ReadStringsFromPubSub, self).__init__()
+if topic and subscription:
+  raise ValueError("Only one of topic or subscription should be provided.")
+
+if not (topic or subscription):
+  raise ValueError("Either a topic or subscription must be provided.")
+
 self._source = _PubSubPayloadSource(
 topic,
 subscription=subscription,
@@ -90,9 +98,11 @@ class _PubSubPayloadSource(dataflow_io.NativeSource):
   """Source for the payload of a message as bytes from a Cloud Pub/Sub topic.
 
   Attributes:
-topic: Cloud Pub/Sub topic in the form "/topics//".
-subscription: Optional existing Cloud Pub/Sub subscription to use in the
-  form "projects//subscriptions/".
+topic: Cloud Pub/Sub topic in the form "/topics//". If
+  provided then topic must be None.
+subscription: Existing Cloud Pub/Sub subscription to use in the
+  form "projects//subscriptions/". If provided then
+  subscription must be None.
 id_label: The attribute on incoming Pub/Sub messages to use as a unique
   record identifier.  When specified, the value of this attribute (which 
can
   be any string that uniquely identifies the record) will be used for
@@ -101,7 +111,10 @@ class _PubSubPayloadSource(dataflow_io.NativeSource):
   case, deduplication of the stream will be strictly best effort.
   """
 
-  def __init__(self, topic, subscription=None, id_label=None):
+  def __init__(self, topic=None, subscription=None, id_label=None):
+# we are using this coder explicitly for portability reasons of PubsubIO
+# across implementations in languages.
+self.coder = coders.BytesCoder()
 self.topic = topic
 self.subscription = subscription
 self.id_label = id_label
@@ -131,6 +144,9 @@ class _PubSubPayloadSink(dataflow_io.NativeSink):
   """Sink for the payload of a message as bytes to a Cloud Pub/Sub topic."""
 
   def __init__(self, topic):
+# we are using this coder explicitly for portability reasons of PubsubIO
+# across implementations in languages.
+self.coder = coders.BytesCoder()
 self.topic = topic
 
   @property

http://git-wip-us.apache.org/repos/asf/beam/blob/b5852d21/sdks/python/apache_beam/io/gcp/pubsub_test.py
--
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_test.py 

[15/50] [abbrv] beam git commit: Expand all PValues to component PCollections always

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
index 6c385d7..1853248 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
@@ -42,12 +42,12 @@ import org.apache.beam.sdk.values.PCollectionView;
 class StreamingViewOverrides {
   static class StreamingCreatePCollectionViewFactory
   extends SingleInputOutputOverrideFactory<
-  PCollection, PCollectionView, 
CreatePCollectionView> {
+  PCollection, PCollection, CreatePCollectionView> {
 @Override
-public PTransformReplacement
+public PTransformReplacement
 getReplacementTransform(
 AppliedPTransform<
-PCollection, PCollectionView, 
CreatePCollectionView>
+PCollection, PCollection, 
CreatePCollectionView>
 transform) {
   StreamingCreatePCollectionView streamingView =
   new 
StreamingCreatePCollectionView<>(transform.getTransform().getView());
@@ -56,7 +56,7 @@ class StreamingViewOverrides {
 }
 
 private static class StreamingCreatePCollectionView
-extends PTransform {
+extends PTransform {
   private final PCollectionView view;
 
   private StreamingCreatePCollectionView(PCollectionView view) {
@@ -64,7 +64,7 @@ class StreamingViewOverrides {
   }
 
   @Override
-  public PCollectionView expand(PCollection input) {
+  public PCollection expand(PCollection input) {
 return input
 .apply(Combine.globally(new 
Concatenate()).withoutDefaults())
 .apply(ParDo.of(StreamingPCollectionViewWriterFn.create(view, 
input.getCoder(

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index 89dc2d5..53215f6 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -920,15 +920,15 @@ public class DataflowPipelineTranslatorTest implements 
Serializable {
 assertAllStepOutputsHaveUniqueIds(job);
 
 List steps = job.getSteps();
-assertEquals(5, steps.size());
+assertEquals(9, steps.size());
 
 @SuppressWarnings("unchecked")
 List> toIsmRecordOutputs =
-(List>) 
steps.get(3).getProperties().get(PropertyNames.OUTPUT_INFO);
+(List>) 
steps.get(7).getProperties().get(PropertyNames.OUTPUT_INFO);
 assertTrue(
 Structs.getBoolean(Iterables.getOnlyElement(toIsmRecordOutputs), 
"use_indexed_format"));
 
-Step collectionToSingletonStep = steps.get(4);
+Step collectionToSingletonStep = steps.get(8);
 assertEquals("CollectionToSingleton", collectionToSingletonStep.getKind());
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index 64aa35a..ac5e0cd 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -508,50 +508,6 @@ public final class TransformTranslator {
 };
   }
 
-  private static  TransformEvaluator viewAsSingleton() 
{
-return new TransformEvaluator() {
-  @Override
-  public void evaluate(View.AsSingleton 

[02/50] [abbrv] beam git commit: This closes #2286

[kenn]

This closes #2286


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/609016d7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/609016d7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/609016d7

Branch: refs/heads/gearpump-runner
Commit: 609016d700c84800cf942482fb7cd2ddaa420b00
Parents: b2de3db 8f4fa43
Author: Aljoscha Krettek 
Authored: Wed Jun 7 19:43:19 2017 +0200
Committer: Aljoscha Krettek 
Committed: Wed Jun 7 19:43:19 2017 +0200

--
 .../translation/types/CoderTypeSerializer.java  |  41 ++-
 .../streaming/io/UnboundedSourceWrapper.java|   2 +
 .../flink/streaming/TestCountingSource.java |  48 ++-
 .../streaming/UnboundedSourceWrapperTest.java   | 309 +++
 4 files changed, 254 insertions(+), 146 deletions(-)
--

[08/50] [abbrv] beam git commit: Remove support for NativeSinks from the Python DirectRunner

[kenn]

Remove support for NativeSinks from the Python DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f2e30886
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f2e30886
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f2e30886

Branch: refs/heads/gearpump-runner
Commit: f2e3088633fef10f19bfd11ff9b508930916a740
Parents: 32f22b7
Author: Charles Chen 
Authored: Wed Jun 7 17:00:57 2017 -0700
Committer: Charles Chen 
Committed: Wed Jun 7 17:01:33 2017 -0700

--
 .../runners/direct/transform_evaluator.py   | 62 +---
 1 file changed, 1 insertion(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f2e30886/sdks/python/apache_beam/runners/direct/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py 
b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index b1cb626..0fec8b8 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -29,7 +29,6 @@ from apache_beam.runners.common import DoFnRunner
 from apache_beam.runners.common import DoFnState
 from apache_beam.runners.direct.watermark_manager import WatermarkManager
 from apache_beam.runners.direct.transform_result import TransformResult
-from apache_beam.runners.dataflow.native_io.iobase import _NativeWrite  # 
pylint: disable=protected-access
 from apache_beam.transforms import core
 from apache_beam.transforms.window import GlobalWindows
 from apache_beam.transforms.window import WindowedValue
@@ -54,7 +53,6 @@ class TransformEvaluatorRegistry(object):
 core.Flatten: _FlattenEvaluator,
 core.ParDo: _ParDoEvaluator,
 core._GroupByKeyOnly: _GroupByKeyOnlyEvaluator,
-_NativeWrite: _NativeWriteEvaluator,
 }
 
   def for_application(
@@ -98,8 +96,7 @@ class TransformEvaluatorRegistry(object):
 Returns:
   True if executor should execute applied_ptransform serially.
 """
-return isinstance(applied_ptransform.transform,
-  (core._GroupByKeyOnly, _NativeWrite))
+return isinstance(applied_ptransform.transform, core._GroupByKeyOnly)
 
 
 class _TransformEvaluator(object):
@@ -403,60 +400,3 @@ class _GroupByKeyOnlyEvaluator(_TransformEvaluator):
 
 return TransformResult(
 self._applied_ptransform, bundles, state, None, None, hold)
-
-
-class _NativeWriteEvaluator(_TransformEvaluator):
-  """TransformEvaluator for _NativeWrite transform."""
-
-  def __init__(self, evaluation_context, applied_ptransform,
-   input_committed_bundle, side_inputs, scoped_metrics_container):
-assert not side_inputs
-super(_NativeWriteEvaluator, self).__init__(
-evaluation_context, applied_ptransform, input_committed_bundle,
-side_inputs, scoped_metrics_container)
-
-assert applied_ptransform.transform.sink
-self._sink = applied_ptransform.transform.sink
-
-  @property
-  def _is_final_bundle(self):
-return (self._execution_context.watermarks.input_watermark
-== WatermarkManager.WATERMARK_POS_INF)
-
-  @property
-  def _has_already_produced_output(self):
-return (self._execution_context.watermarks.output_watermark
-== WatermarkManager.WATERMARK_POS_INF)
-
-  def start_bundle(self):
-# state: [values]
-self.state = (self._execution_context.existing_state
-  if self._execution_context.existing_state else [])
-
-  def process_element(self, element):
-self.state.append(element)
-
-  def finish_bundle(self):
-# finish_bundle will append incoming bundles in memory until all the 
bundles
-# carrying data is processed. This is done to produce only a single output
-# shard (some tests depends on this behavior). It is possible to have
-# incoming empty bundles after the output is produced, these bundles will 
be
-# ignored and would not generate additional output files.
-# TODO(altay): Do not wait until the last bundle to write in a single 
shard.
-if self._is_final_bundle:
-  if self._has_already_produced_output:
-# Ignore empty bundles that arrive after the output is produced.
-assert self.state == []
-  else:
-self._sink.pipeline_options = self._evaluation_context.pipeline_options
-with self._sink.writer() as writer:
-  for v in self.state:
-writer.Write(v.value)
-  state = None
-  hold = WatermarkManager.WATERMARK_POS_INF
-else:
-  state = self.state
-  hold = WatermarkManager.WATERMARK_NEG_INF
-
-return TransformResult(
-self._applied_ptransform, [], state, None, None, hold)

[GitHub] beam pull request #3292: [BEAM-79] Merge master into gearpump-runner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[45/50] [abbrv] beam git commit: Port DirectGroupByKey to SDK-agnostic APIs

[kenn]

Port DirectGroupByKey to SDK-agnostic APIs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/02dbaefd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/02dbaefd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/02dbaefd

Branch: refs/heads/gearpump-runner
Commit: 02dbaefd2bbad0f0ff0b87469d184137b220fae7
Parents: 8c5b57e
Author: Kenneth Knowles 
Authored: Fri May 26 14:27:23 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 19:56:52 2017 -0700

--
 .../apache/beam/runners/direct/DirectGroupByKey.java  | 13 +++--
 .../direct/DirectGroupByKeyOverrideFactory.java   | 14 +++---
 2 files changed, 18 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/02dbaefd/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
index 2fc0dd4..06b8e29 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
@@ -36,13 +36,17 @@ import org.apache.beam.sdk.values.WindowingStrategy;
 
 class DirectGroupByKey
 extends ForwardingPTransform>, PCollection>> {
-  private final GroupByKey original;
+  private final PTransform>, PCollection>> original;
 
   static final String DIRECT_GBKO_URN = 
"urn:beam:directrunner:transforms:gbko:v1";
   static final String DIRECT_GABW_URN = 
"urn:beam:directrunner:transforms:gabw:v1";
+  private final WindowingStrategy outputWindowingStrategy;
 
-  DirectGroupByKey(GroupByKey from) {
-this.original = from;
+  DirectGroupByKey(
+  PTransform>, PCollection>> 
original,
+  WindowingStrategy outputWindowingStrategy) {
+this.original = original;
+this.outputWindowingStrategy = outputWindowingStrategy;
   }
 
   @Override
@@ -57,9 +61,6 @@ class DirectGroupByKey
 // key/value input elements and the window merge operation of the
 // window function associated with the input PCollection.
 WindowingStrategy inputWindowingStrategy = 
input.getWindowingStrategy();
-// Update the windowing strategy as appropriate.
-WindowingStrategy outputWindowingStrategy =
-original.updateWindowingStrategy(inputWindowingStrategy);
 
 // By default, implement GroupByKey via a series of lower-level operations.
 return input

http://git-wip-us.apache.org/repos/asf/beam/blob/02dbaefd/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
index c2eb5e7..9c2de3d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
@@ -17,26 +17,34 @@
  */
 package org.apache.beam.runners.direct;
 
+import com.google.common.collect.Iterables;
 import org.apache.beam.runners.core.construction.PTransformReplacements;
 import 
org.apache.beam.runners.core.construction.SingleInputOutputOverrideFactory;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
 /** A {@link PTransformOverrideFactory} for {@link GroupByKey} PTransforms. */
 final class DirectGroupByKeyOverrideFactory
 extends SingleInputOutputOverrideFactory<
-PCollection>, PCollection>, GroupByKey> {
+PCollection>, PCollection>,
+PTransform>, PCollection>, PCollection>>
   getReplacementTransform(
   AppliedPTransform<
-  PCollection>, PCollection>, 
GroupByKey>
+  PCollection>, PCollection>,
+  

[49/50] [abbrv] beam git commit: Fix side input handling in DoFnFunction

[kenn]

Fix side input handling in DoFnFunction


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7653e7ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7653e7ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7653e7ed

Branch: refs/heads/gearpump-runner
Commit: 7653e7ed6de3d9db822dcd390d2bf70819954fa5
Parents: 98854d4
Author: manuzhang 
Authored: Wed Jun 7 14:08:04 2017 +0800
Committer: manuzhang 
Committed: Mon Jun 12 11:45:37 2017 +0800

--
 .../translators/TranslationContext.java |  2 ++
 .../translators/functions/DoFnFunction.java | 23 
 .../gearpump/translators/io/GearpumpSource.java |  1 -
 .../translators/utils/TranslatorUtils.java  |  5 ++---
 4 files changed, 8 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7653e7ed/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
index 4090354..64a1e0d 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
@@ -68,6 +68,8 @@ public class TranslationContext {
   public  void setOutputStream(PValue output, JavaStream 
outputStream) {
 if (!streams.containsKey(output)) {
   streams.put(output, outputStream);
+} else {
+  throw new RuntimeException("set stream for duplicated output " + output);
 }
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7653e7ed/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
index f521d7b..6e4fbeb 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
@@ -24,7 +24,6 @@ import com.google.common.collect.Lists;
 
 import java.io.Serializable;
 import java.util.Collection;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedList;
@@ -66,7 +65,6 @@ public class DoFnFunction extends
   private transient PushbackSideInputDoFnRunner doFnRunner;
   private transient SideInputHandler sideInputReader;
   private transient List pushedBackValues;
-  private transient Map>> 
sideInputValues;
   private final Collection sideInputs;
   private final Map tagsToSideInputs;
   private final TupleTag mainOutput;
@@ -109,7 +107,6 @@ public class DoFnFunction extends
 doFnRunner = doFnRunnerFactory.createRunner(sideInputReader);
 
 pushedBackValues = new LinkedList<>();
-sideInputValues = new HashMap<>();
 outputManager.setup(mainOutput, sideOutputs);
   }
 
@@ -132,25 +129,14 @@ public class DoFnFunction extends
   } else {
 // side input
 PCollectionView sideInput = 
tagsToSideInputs.get(unionValue.getUnionTag());
-WindowedValue sideInputValue =
-(WindowedValue) unionValue.getValue();
-Object value = sideInputValue.getValue();
-if (!(value instanceof Iterable)) {
-  sideInputValue = sideInputValue.withValue(Lists.newArrayList(value));
-}
-if (!sideInputValues.containsKey(sideInput)) {
-  sideInputValues.put(sideInput, new 
LinkedList>());
-}
-sideInputValues.get(sideInput).add((WindowedValue) 
sideInputValue);
+WindowedValue sideInputValue =
+(WindowedValue) unionValue.getValue();
+sideInputReader.addSideInputValue(sideInput, sideInputValue);
   }
 }
 
+
 for (PCollectionView sideInput: sideInputs) {
-  if (sideInputValues.containsKey(sideInput)) {
-for (WindowedValue value: sideInputValues.get(sideInput)) 
{
-  sideInputReader.addSideInputValue(sideInput, value);
-}
-  }
   for (WindowedValue value : 

[44/50] [abbrv] beam git commit: Port ViewOverrideFactory to SDK-agnostic APIs

[kenn]

Port ViewOverrideFactory to SDK-agnostic APIs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8c5b57ea
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8c5b57ea
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8c5b57ea

Branch: refs/heads/gearpump-runner
Commit: 8c5b57ea8445cd50a35c6dffb460dcf0f426e700
Parents: b4c7716
Author: Kenneth Knowles 
Authored: Fri May 26 14:26:55 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 19:56:52 2017 -0700

--
 .../CreatePCollectionViewTranslation.java   |  4 +-
 .../runners/direct/ViewOverrideFactory.java | 48 
 .../direct/ViewEvaluatorFactoryTest.java|  3 +-
 .../runners/direct/ViewOverrideFactoryTest.java | 23 --
 .../beam/sdk/values/PCollectionViews.java   | 10 
 5 files changed, 62 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8c5b57ea/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
index aa24909..8fc99b9 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
@@ -56,8 +56,8 @@ public class CreatePCollectionViewTranslation {
   @Deprecated
   public static  PCollectionView getView(
   AppliedPTransform<
-  PCollection, PCollectionView,
-  PTransform>
+  PCollection, PCollection,
+  PTransform>
   application)
   throws IOException {
 

http://git-wip-us.apache.org/repos/asf/beam/blob/8c5b57ea/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
index 06a7388..5dcf016 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
@@ -18,8 +18,9 @@
 
 package org.apache.beam.runners.direct;
 
+import java.io.IOException;
 import java.util.Map;
-import org.apache.beam.runners.core.construction.ForwardingPTransform;
+import 
org.apache.beam.runners.core.construction.CreatePCollectionViewTranslation;
 import org.apache.beam.runners.core.construction.PTransformReplacements;
 import 
org.apache.beam.runners.core.construction.PTransformTranslation.RawPTransform;
 import org.apache.beam.runners.core.construction.ReplacementOutputs;
@@ -43,16 +44,30 @@ import org.apache.beam.sdk.values.TupleTag;
  */
 class ViewOverrideFactory
 implements PTransformOverrideFactory<
-PCollection, PCollection, CreatePCollectionView> {
+PCollection, PCollection,
+PTransform> {
 
   @Override
   public PTransformReplacement 
getReplacementTransform(
   AppliedPTransform<
-  PCollection, PCollection, 
CreatePCollectionView>
+  PCollection, PCollection,
+  PTransform>
   transform) {
-return PTransformReplacement.of(
+
+PCollectionView view;
+try {
+  view = CreatePCollectionViewTranslation.getView(transform);
+} catch (IOException exc) {
+  throw new RuntimeException(
+  String.format(
+  "Could not extract %s from transform %s",
+  PCollectionView.class.getSimpleName(), transform),
+  exc);
+}
+
+  return PTransformReplacement.of(
 PTransformReplacements.getSingletonMainInput(transform),
-new GroupAndWriteView<>(transform.getTransform()));
+new GroupAndWriteView(view));
   }
 
   @Override
@@ -63,11 +78,11 @@ class ViewOverrideFactory
 
   /** The {@link DirectRunner} composite override for {@link 
CreatePCollectionView}. */
   static class GroupAndWriteView
-  extends ForwardingPTransform {
-private 

[50/50] [abbrv] beam git commit: This closes #3292: Merge master into gearpump-runner

[kenn]

This closes #3292: Merge master into gearpump-runner

Adjustments to gearpump-runner:

  Fix side input handling in DoFnFunction
  Respect WindowFn#getOutputTime in gearpump-runner
  Activate Gearpump local-validates-runner-tests in precommit
  Update against master changes

Merged from master:

  [BEAM-972] Add more unit test to Gearpump runner
  [BEAM-972] Add unit tests to Gearpump runner
  [BEAM-79] Fix gearpump-runner merge conflicts and test failure
  enable ParDoTest
  [BEAM-79] Add SideInput support for GearpumpRunner
  [BEAM-79] Support merging windows in GearpumpRunner
  [BEAM-79] Fix PostCommit test confs for Gearpump runner
  note thread is interrupted on InterruptedException
  Remove cache for Gearpump on travis
  reduce timeout to wait for result
  fix ParDo.BoundMulti translation
  return encoded key for GroupByKey translation
  support OutputTimeFn
  update to latest gearpump dsl function interface
  fix group by window
  activate ROS on Gearpump by default
  update ROS configurations
  [BEAM-1180] Implement GearpumpPipelineResult
  [BEAM-79] Upgrade to beam-0.5.0-incubating-SNAPSHOT
  [BEAM-79] Update to latest Gearpump API
  Fix NoOpAggregatorFactory
  Remove print to stdout
  Skip window assignment when windows don't change
  Add Window.Bound translator
  Upgrade Gearpump version
  [BEAM-79] fix gearpump runner build failure
  [BEAM-79] update GearpumpPipelineResult
  [BEAM-79] Port Gearpump runner from OldDoFn to new DoFn
  upgrade gearpump-runner to 0.4.0-incubating-SNAPSHOT
  remove "pipeline" in runner name
  post-merge fix
  [BEAM-79] fix integration-test failure
  fix import order
  !fixup Minor javadoc clean-up
  Added even more javadoc to TextIO#withHeader and TextIO#withFooter (2).
  Added even more javadoc to TextIO#withHeader and TextIO#withFooter.
  Added javadoc to TextIO#withHeader and TextIO#withFooter.
  Reverted header and footer to be of type String.
  Revised according to comments following a code review.
  Add header/footer support to TextIO.Write
  [BEAM-242] Enable and fix checkstyle in Flink runner examples
  Remove timeout in JAXBCoderTest
  Be more accepting in UnboundedReadDeduplicatorTest
  BigQuery: limit max job polling time to 1 minute
  [BEAM-242] Enable checkstyle and fix checkstyle errors in Flink runner
  [BEAM-456] Add MongoDbIO
  FluentBackoff: a replacement for a variety of custom backoff implementations
  Remove the DataflowRunner instructions from examples
  Put classes in runners-core package into runners.core namespace
  Delegate populateDipslayData to wrapped combineFn's
  Fixed Combine display data
  Cloud Datastore naming clean-up
  DatastoreIO SplitQueryFn integration test
  Add Latest CombineFn and PTransforms
  Remove empty unused method in TestStreamEvaluatorFactory
  Test that multiple instances of TestStream are supported
  Correct some accidental renames
  Fix condition in FlinkStreamingPipelineTranslator
  Address comments of Flink Side-Input PR
  [BEAM-569] Define maxNumRecords default value to Long.MAX_VALUE in JmsIO
  Add LeaderBoardTest
  take advantage of setup/teardown for KafkaWriter
  Returned KafkaIO getWatermark log line in debug mode
  [BEAM-572] Remove Spark Reference in WordCount
  Update Dataflow Container Version
  [BEAM-313] Provide a context for SparkRunner
  DataflowRunner: get PBegin from PInput
  [BEAM-592] Fix SparkRunner Dependency Problem in WordCount
  Fix javadoc in Kinesis
  Organize imports in Kinesis
  kinesis: a connector for Amazon Kinesis
  [BEAM-589] Fixing IO.Read transformation
  Query latest timestamp
  travis.yml: disable updating snapshots
  Added support for reporting aggregator values to Spark sinks
  [BEAM-294] Rename dataflow references to beam
  Modified BigtableIO to use DoFn setup/tearDown methods instead of 
startBundle/finishBundle
  checkstyle: prohibit API client repackaged Guava
  Make WriteTest more resilient to Randomness
  Update DoFn javadocs to remove references to OldDoFn and Dataflow
  [BEAM-545] Promote JobName to PipelineOptions
  Move the samples data to gs://apache-beam-samples/
  Cleanup some javadoc that referring Dataflow
  BigQueryIO.Write: raise size limit to 11 TiB
  Optimize imports
  Update checkstyle.xml to put all imports in one group
  Fix Exception Unwrapping in TestFlinkRunner
  Make ParDoLifecycleTest Serializable to Fix Test with TupleTag
  Use AllPanes as the PaneExtractor in IterableAssert
  Fix combine tests with Accumulation Mode
  Enable Flink Streaming Runner RunnableOnService tests
  Fix Flink Runner Pom for Batch RunnableOnService tests
  Don't Suppress Throwable in PAssert in Streaming Mode
  ...


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/559e3c34
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/559e3c34
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/559e3c34

Branch: refs/heads/gearpump-runner
Commit: 559e3c341f879cc06f823363ed0fa4c1eab8a6a5
Parents: 1597f3c 

[39/50] [abbrv] beam git commit: Closes #3332

[kenn]

Closes #3332


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d2c4093e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d2c4093e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d2c4093e

Branch: refs/heads/gearpump-runner
Commit: d2c4093ed2fd47e28b31898a17e28d0575aa9ff9
Parents: 97957ea fb61c54
Author: Robert Bradshaw 
Authored: Fri Jun 9 13:00:55 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 9 13:00:55 2017 -0700

--
 sdks/python/apache_beam/io/gcp/pubsub.py  | 16 +++-
 sdks/python/apache_beam/io/gcp/pubsub_test.py | 10 --
 2 files changed, 3 insertions(+), 23 deletions(-)
--

[48/50] [abbrv] beam git commit: This closes #3338: [BEAM-2371] Port some DirectRunner overrides to SDK-agnostic APIs

[kenn]

This closes #3338: [BEAM-2371] Port some DirectRunner overrides to SDK-agnostic 
APIs

  Port DirectRunner TestStream override to SDK-agnostic APIs
  Port DirectRunner WriteFiles override to SDK-agnostic APIs
  Port DirectGroupByKey to SDK-agnostic APIs
  Port ViewOverrideFactory to SDK-agnostic APIs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1597f3ca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1597f3ca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1597f3ca

Branch: refs/heads/gearpump-runner
Commit: 1597f3ca64558f0099237aeb618b144e132ddcc6
Parents: b4c7716 eaaf45f
Author: Kenneth Knowles 
Authored: Fri Jun 9 19:57:17 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 19:57:17 2017 -0700

--
 .../CreatePCollectionViewTranslation.java   |  4 +-
 .../core/construction/PTransformMatchers.java   | 17 +--
 .../construction/TestStreamTranslation.java | 49 +++-
 .../beam/runners/direct/DirectGroupByKey.java   | 13 +++---
 .../direct/DirectGroupByKeyOverrideFactory.java | 14 --
 .../direct/TestStreamEvaluatorFactory.java  | 20 ++--
 .../runners/direct/ViewOverrideFactory.java | 48 +++
 .../direct/WriteWithShardingFactory.java| 30 
 .../direct/ViewEvaluatorFactoryTest.java|  3 +-
 .../runners/direct/ViewOverrideFactoryTest.java | 23 +++--
 .../direct/WriteWithShardingFactoryTest.java| 26 +++
 .../org/apache/beam/sdk/testing/TestStream.java | 12 +
 .../beam/sdk/values/PCollectionViews.java   | 10 
 13 files changed, 207 insertions(+), 62 deletions(-)
--

[17/50] [abbrv] beam git commit: Avoid flakiness in data channel for empty streams.

[kenn]

Avoid flakiness in data channel for empty streams.

As empty stream is used as end-of-stream marker, don't ever send
it as the data itself.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4ebebfdb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4ebebfdb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4ebebfdb

Branch: refs/heads/gearpump-runner
Commit: 4ebebfdb34de3e209c033de15e32cf67ab346d44
Parents: e066a9d
Author: Robert Bradshaw 
Authored: Wed Jun 7 23:00:43 2017 -0700
Committer: Luke Cwik 
Committed: Thu Jun 8 10:06:17 2017 -0700

--
 .../python/apache_beam/runners/worker/data_plane.py | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4ebebfdb/sdks/python/apache_beam/runners/worker/data_plane.py
--
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py 
b/sdks/python/apache_beam/runners/worker/data_plane.py
index 5edd0b4..7365db6 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -167,12 +167,18 @@ class _GrpcDataChannel(DataChannel):
 yield data
 
   def output_stream(self, instruction_id, target):
+# TODO: Return an output stream that sends data
+# to the Runner once a fixed size buffer is full.
+# Currently we buffer all the data before sending
+# any messages.
 def add_to_send_queue(data):
-  self._to_send.put(
-  beam_fn_api_pb2.Elements.Data(
-  instruction_reference=instruction_id,
-  target=target,
-  data=data))
+  if data:
+self._to_send.put(
+beam_fn_api_pb2.Elements.Data(
+instruction_reference=instruction_id,
+target=target,
+data=data))
+  # End of stream marker.
   self._to_send.put(
   beam_fn_api_pb2.Elements.Data(
   instruction_reference=instruction_id,

[20/50] [abbrv] beam git commit: This closes #3322

[kenn]

This closes #3322


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fdfc70ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fdfc70ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fdfc70ee

Branch: refs/heads/gearpump-runner
Commit: fdfc70eecb8a3833c155e8bb7b6e128d956a61de
Parents: d81ed21 3e04902
Author: Ahmet Altay 
Authored: Thu Jun 8 10:55:47 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:55:47 2017 -0700

--
 .../runners/direct/watermark_manager.py | 20 +---
 sdks/python/apache_beam/utils/timestamp.py  |  5 +
 2 files changed, 22 insertions(+), 3 deletions(-)
--

[18/50] [abbrv] beam git commit: Avoid flakiness in data channel for empty streams.

[kenn]

Avoid flakiness in data channel for empty streams.

This closes #3324


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d81ed217
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d81ed217
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d81ed217

Branch: refs/heads/gearpump-runner
Commit: d81ed21726cb50962bc4fcfb57d50e11eacdcf3b
Parents: e066a9d 4ebebfd
Author: Luke Cwik 
Authored: Thu Jun 8 10:09:12 2017 -0700
Committer: Luke Cwik 
Committed: Thu Jun 8 10:09:12 2017 -0700

--
 .../python/apache_beam/runners/worker/data_plane.py | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--

[22/50] [abbrv] beam git commit: This closes #3302

[kenn]

This closes #3302


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b53e6f0d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b53e6f0d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b53e6f0d

Branch: refs/heads/gearpump-runner
Commit: b53e6f0dc91948040364d1e742b023d5a2be8433
Parents: fdfc70e 156f326
Author: Ahmet Altay 
Authored: Thu Jun 8 10:57:11 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:57:11 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java  | 16 +---
 2 files changed, 10 insertions(+), 8 deletions(-)
--

[03/50] [abbrv] beam git commit: [BEAM-2421] Swap to use an Impulse primitive + DoFn for Create when executing with the Fn API.

[kenn]

[BEAM-2421] Swap to use an Impulse primitive + DoFn for Create when executing 
with the Fn API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1cdb80cb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1cdb80cb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1cdb80cb

Branch: refs/heads/gearpump-runner
Commit: 1cdb80cb6319c04fa94961c14c038a5e15736d68
Parents: 5f7e73b
Author: Luke Cwik 
Authored: Wed Jun 7 08:53:14 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jun 7 13:41:20 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 158 +--
 1 file changed, 145 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1cdb80cb/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index ed29330..3e7c8ce 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -49,6 +49,7 @@ import java.net.URLClassLoader;
 import java.nio.channels.Channels;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
@@ -57,6 +58,7 @@ import java.util.Random;
 import java.util.Set;
 import java.util.SortedSet;
 import java.util.TreeSet;
+import org.apache.beam.runners.core.construction.CoderTranslation;
 import org.apache.beam.runners.core.construction.DeduplicatedFlattenFactory;
 import org.apache.beam.runners.core.construction.EmptyFlattenAsCreateFactory;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
@@ -79,10 +81,12 @@ import org.apache.beam.sdk.Pipeline.PipelineVisitor;
 import org.apache.beam.sdk.PipelineResult.State;
 import org.apache.beam.sdk.PipelineRunner;
 import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
 import org.apache.beam.sdk.extensions.gcp.storage.PathValidator;
 import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.FileSystems;
@@ -103,6 +107,7 @@ import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.runners.TransformHierarchy.Node;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.Combine.GroupedValues;
+import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
@@ -113,6 +118,7 @@ import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.WithKeys;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.InstanceBuilder;
 import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.util.NameUtils;
@@ -312,6 +318,12 @@ public class DataflowRunner extends 
PipelineRunner {
 PTransformMatchers.classEqualTo(PubsubUnboundedSink.class),
 new StreamingPubsubIOWriteOverrideFactory(this)));
   }
+  if (hasExperiment(options, "beam_fn_api")) {
+overridesBuilder.add(
+PTransformOverride.of(
+PTransformMatchers.classEqualTo(Create.Values.class),
+new StreamingFnApiCreateOverrideFactory()));
+  }
   overridesBuilder
   .add(
   // Streaming Bounded Read is implemented in terms of Streaming 
Unbounded Read, and
@@ -428,15 +440,12 @@ public class DataflowRunner extends 
PipelineRunner {
 public PTransformReplacement 
getReplacementTransform(
 AppliedPTransform> transform) {
   PTransform original = transform.getTransform();
-  PCollection output =
-  (PCollection) 
Iterables.getOnlyElement(transform.getOutputs().values());
   return PTransformReplacement.of(
   transform.getPipeline().begin(),
   InstanceBuilder.ofType(replacement)
   

[35/50] [abbrv] beam git commit: [BEAM-2410] This closes #3328

[kenn]

[BEAM-2410] This closes #3328


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1c6861f2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1c6861f2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1c6861f2

Branch: refs/heads/gearpump-runner
Commit: 1c6861f22249dadacb34994564d33b14f1d57d3d
Parents: e980ae9 7caea7a
Author: Jean-Baptiste Onofré 
Authored: Fri Jun 9 07:31:27 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Fri Jun 9 07:31:27 2017 +0200

--
 .../sdk/io/common/IOTestPipelineOptions.java|  6 +-
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |  4 +-
 .../elasticsearch/ElasticSearchIOTestUtils.java | 81 +++-
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java | 14 ++--
 .../io/elasticsearch/ElasticsearchIOTest.java   | 36 +
 .../elasticsearch/ElasticsearchTestDataSet.java | 37 -
 6 files changed, 87 insertions(+), 91 deletions(-)
--

[24/50] [abbrv] beam git commit: Add Runner API oriented PTransformMatchers for DirectRunner overrides

[kenn]

Add Runner API oriented PTransformMatchers for DirectRunner overrides


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ca7b9c28
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ca7b9c28
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ca7b9c28

Branch: refs/heads/gearpump-runner
Commit: ca7b9c288151d318898ab000b91d26fcf62046ca
Parents: b53e6f0
Author: Kenneth Knowles 
Authored: Thu May 25 06:29:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:36:28 2017 -0700

--
 .../core/construction/PTransformMatchers.java   | 94 +++-
 .../construction/PTransformTranslation.java |  7 +-
 .../construction/PTransformMatchersTest.java| 32 +++
 3 files changed, 128 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ca7b9c28/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
index bfe24a0..c339891 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
@@ -18,6 +18,7 @@
 package org.apache.beam.runners.core.construction;
 
 import com.google.common.base.MoreObjects;
+import java.io.IOException;
 import java.util.HashSet;
 import java.util.Set;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -50,6 +51,34 @@ public class PTransformMatchers {
   private PTransformMatchers() {}
 
   /**
+   * Returns a {@link PTransformMatcher} that matches a {@link PTransform} if 
the URN of the
+   * {@link PTransform} is equal to the URN provided ot this matcher.
+   */
+  public static PTransformMatcher urnEqualTo(String urn) {
+return new EqualUrnPTransformMatcher(urn);
+  }
+
+  private static class EqualUrnPTransformMatcher implements PTransformMatcher {
+private final String urn;
+
+private EqualUrnPTransformMatcher(String urn) {
+  this.urn = urn;
+}
+
+@Override
+public boolean matches(AppliedPTransform application) {
+  return 
urn.equals(PTransformTranslation.urnForTransformOrNull(application.getTransform()));
+}
+
+@Override
+public String toString() {
+  return MoreObjects.toStringHelper(this)
+  .add("urn", urn)
+  .toString();
+}
+  }
+
+  /**
* Returns a {@link PTransformMatcher} that matches a {@link PTransform} if 
the class of the
* {@link PTransform} is equal to the {@link Class} provided ot this matcher.
*/
@@ -151,6 +180,68 @@ public class PTransformMatchers {
   }
 
   /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo} by URN if it has 
a splittable {@link
+   * DoFn}.
+   */
+  public static PTransformMatcher splittableParDo() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+if (PTransformTranslation.PAR_DO_TRANSFORM_URN.equals(
+
PTransformTranslation.urnForTransformOrNull(application.getTransform( {
+
+  try {
+return ParDoTranslation.isSplittable(application);
+  } catch (IOException e) {
+throw new RuntimeException(
+String.format(
+"Transform with URN %s could not be translated",
+PTransformTranslation.PAR_DO_TRANSFORM_URN),
+e);
+  }
+}
+return false;
+  }
+
+  @Override
+  public String toString() {
+return 
MoreObjects.toStringHelper("SplittableParDoMultiMatcher").toString();
+  }
+};
+  }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo} transform by URN
+   * and whether it contains state or timers as specified by {@link 
ParDoTranslation}.
+   */
+  public static PTransformMatcher stateOrTimerParDo() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+if (PTransformTranslation.PAR_DO_TRANSFORM_URN.equals(
+
PTransformTranslation.urnForTransformOrNull(application.getTransform( {
+
+  try {
+return ParDoTranslation.usesStateOrTimers(application);
+  } catch (IOException e) {
+throw new RuntimeException(
+String.format(
+"Transform with URN %s could not be translated",
+

[10/50] [abbrv] beam git commit: Make BytesCoder to be a known type

[kenn]

Make BytesCoder to be a known type


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d94ac58e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d94ac58e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d94ac58e

Branch: refs/heads/gearpump-runner
Commit: d94ac58ea2d12f55743e8ad27a02bdb83c194da7
Parents: 66460cb
Author: Vikas Kedigehalli 
Authored: Wed Jun 7 16:26:21 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jun 7 20:05:40 2017 -0700

--
 sdks/python/apache_beam/coders/coders.py  | 5 +
 sdks/python/apache_beam/runners/worker/operation_specs.py | 4 
 2 files changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d94ac58e/sdks/python/apache_beam/coders/coders.py
--
diff --git a/sdks/python/apache_beam/coders/coders.py 
b/sdks/python/apache_beam/coders/coders.py
index f40045d..f3e0b43 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -286,6 +286,11 @@ class BytesCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def as_cloud_object(self):
+return {
+'@type': 'kind:bytes',
+}
+
   def __eq__(self, other):
 return type(self) == type(other)
 

http://git-wip-us.apache.org/repos/asf/beam/blob/d94ac58e/sdks/python/apache_beam/runners/worker/operation_specs.py
--
diff --git a/sdks/python/apache_beam/runners/worker/operation_specs.py 
b/sdks/python/apache_beam/runners/worker/operation_specs.py
index db5eb76..b8d19a1 100644
--- a/sdks/python/apache_beam/runners/worker/operation_specs.py
+++ b/sdks/python/apache_beam/runners/worker/operation_specs.py
@@ -339,6 +339,10 @@ def get_coder_from_spec(coder_spec):
 assert len(coder_spec['component_encodings']) == 1
 return coders.coders.LengthPrefixCoder(
 get_coder_from_spec(coder_spec['component_encodings'][0]))
+  elif coder_spec['@type'] == 'kind:bytes':
+assert ('component_encodings' not in coder_spec
+or len(coder_spec['component_encodings'] == 0))
+return coders.BytesCoder()
 
   # We pass coders in the form "$" to make the job
   # description JSON more readable.

[01/50] [abbrv] beam git commit: [BEAM-1779] Port UnboundedSourceWrapperTest to use Flink operator test harness

[kenn]

Repository: beam
Updated Branches:
  refs/heads/gearpump-runner 99221e739 -> 559e3c341


[BEAM-1779] Port UnboundedSourceWrapperTest to use Flink operator test harness


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c1dc8f53
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c1dc8f53
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c1dc8f53

Branch: refs/heads/gearpump-runner
Commit: c1dc8f53c5438b575a7e84e9f680616ead49d61e
Parents: 62b942a
Author: Aljoscha Krettek 
Authored: Wed Mar 22 11:43:30 2017 +0100
Committer: Aljoscha Krettek 
Committed: Wed Jun 7 19:43:11 2017 +0200

--
 .../flink/streaming/TestCountingSource.java |  48 +++--
 .../streaming/UnboundedSourceWrapperTest.java   | 198 +++
 2 files changed, 110 insertions(+), 136 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c1dc8f53/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TestCountingSource.java
--
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TestCountingSource.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TestCountingSource.java
index 3a08088..edf548a 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TestCountingSource.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TestCountingSource.java
@@ -133,18 +133,8 @@ public class TestCountingSource
   public Coder getCheckpointMarkCoder() {
 return DelegateCoder.of(
 VarIntCoder.of(),
-new DelegateCoder.CodingFunction() {
-  @Override
-  public Integer apply(CounterMark input) {
-return input.current;
-  }
-},
-new DelegateCoder.CodingFunction() {
-  @Override
-  public CounterMark apply(Integer input) {
-return new CounterMark(input);
-  }
-});
+new FromCounterMark(),
+new ToCounterMark());
   }
 
   @Override
@@ -251,4 +241,38 @@ public class TestCountingSource
   public Coder> getDefaultOutputCoder() {
 return KvCoder.of(VarIntCoder.of(), VarIntCoder.of());
   }
+
+  private class FromCounterMark implements 
DelegateCoder.CodingFunction {
+@Override
+public Integer apply(CounterMark input) {
+  return input.current;
+}
+
+@Override
+public int hashCode() {
+  return FromCounterMark.class.hashCode();
+}
+
+@Override
+public boolean equals(Object obj) {
+  return obj instanceof FromCounterMark;
+}
+  }
+
+  private class ToCounterMark implements DelegateCoder.CodingFunction {
+@Override
+public CounterMark apply(Integer input) {
+  return new CounterMark(input);
+}
+
+@Override
+public int hashCode() {
+  return ToCounterMark.class.hashCode();
+}
+
+@Override
+public boolean equals(Object obj) {
+  return obj instanceof ToCounterMark;
+}
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/c1dc8f53/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
--
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
index e3875bc..716e71d 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/UnboundedSourceWrapperTest.java
@@ -20,36 +20,20 @@ package org.apache.beam.runners.flink.streaming;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
 
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
-import java.util.Collections;
 import java.util.HashSet;
-import java.util.List;
 import java.util.Set;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper;
 import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.ValueWithRecordId;
-import 

[12/50] [abbrv] beam git commit: Closes #3317

[kenn]

Closes #3317


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/86e1fab6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/86e1fab6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/86e1fab6

Branch: refs/heads/gearpump-runner
Commit: 86e1fab69f296247624d7799f151ff25c3b7243a
Parents: 0a0a1bc b5852d2
Author: Robert Bradshaw 
Authored: Wed Jun 7 22:55:00 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jun 7 22:55:00 2017 -0700

--
 sdks/python/apache_beam/io/gcp/pubsub.py| 32 +++-
 sdks/python/apache_beam/io/gcp/pubsub_test.py   | 28 +++--
 .../runners/dataflow/dataflow_runner.py | 23 ++
 3 files changed, 67 insertions(+), 16 deletions(-)
--

[28/50] [abbrv] beam git commit: Implements HCatalogIO

[kenn]

Implements HCatalogIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/349898c4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/349898c4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/349898c4

Branch: refs/heads/gearpump-runner
Commit: 349898c4702fc3e52d8c0cd1c5a04f14cd40fd27
Parents: 7c5a70e
Author: Seshadri Chakkravarthy 
Authored: Thu May 18 12:07:01 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:05:38 2017 -0700

--
 sdks/java/io/hcatalog/pom.xml   | 163 ++
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 511 +++
 .../beam/sdk/io/hcatalog/package-info.java  |  22 +
 .../io/hcatalog/EmbeddedMetastoreService.java   |  88 
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java| 271 ++
 .../sdk/io/hcatalog/HCatalogIOTestUtils.java| 106 
 .../hcatalog/src/test/resources/hive-site.xml   | 301 +++
 sdks/java/io/pom.xml|   1 +
 8 files changed, 1463 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/349898c4/sdks/java/io/hcatalog/pom.xml
--
diff --git a/sdks/java/io/hcatalog/pom.xml b/sdks/java/io/hcatalog/pom.xml
new file mode 100644
index 000..19b62a5
--- /dev/null
+++ b/sdks/java/io/hcatalog/pom.xml
@@ -0,0 +1,163 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+  4.0.0
+
+  
+org.apache.beam
+beam-sdks-java-io-parent
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  beam-sdks-java-io-hcatalog
+  Apache Beam :: SDKs :: Java :: IO :: HCatalog
+  IO to read and write for HCatalog source.
+
+  
+2.1.0
+2.5
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-surefire-plugin
+
+  true
+
+  
+
+  
+  
+  
+
+  org.apache.beam
+  beam-sdks-java-core
+
+
+
+  org.apache.beam
+  beam-sdks-java-io-hadoop-common
+
+
+
+  org.apache.hadoop
+  hadoop-common
+
+
+
+  commons-io
+  commons-io
+  ${apache.commons.version}
+
+
+
+  org.slf4j
+  slf4j-api
+
+
+
+  com.google.guava
+  guava
+
+
+
+  com.google.code.findbugs
+  jsr305
+
+
+
+  org.apache.hive
+  hive-exec
+  ${hive.version}
+
+
+
+  com.google.auto.value
+  auto-value
+  provided
+
+
+
+  org.apache.hive.hcatalog
+  hive-hcatalog-core
+  ${hive.version}
+  
+
+  org.apache.hive
+  hive-exec
+
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
+  org.apache.hive.hcatalog
+  hive-hcatalog-core
+  tests
+  test
+  ${hive.version}
+
+
+
+  junit
+  junit
+  test
+
+
+
+  org.apache.hive
+  hive-exec
+  ${hive.version}
+  test-jar
+  test
+
+
+
+  org.apache.hive
+  hive-common
+  ${hive.version}
+  test
+
+
+
+  org.apache.hive
+  hive-cli
+  ${hive.version}
+  test
+
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  test
+
+
+
+  org.hamcrest
+  hamcrest-all
+  test
+
+  
+
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/beam/blob/349898c4/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
--
diff --git 
a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
 
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
new file mode 100644
index 000..07b56e3
--- /dev/null
+++ 
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the 

[09/50] [abbrv] beam git commit: This closes #3321

[kenn]

This closes #3321


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/66460cb2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/66460cb2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/66460cb2

Branch: refs/heads/gearpump-runner
Commit: 66460cb2d3db56261483c39acdea4786fec156fd
Parents: 32f22b7 f2e3088
Author: Ahmet Altay 
Authored: Wed Jun 7 18:45:42 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jun 7 18:45:42 2017 -0700

--
 .../runners/direct/transform_evaluator.py   | 62 +---
 1 file changed, 1 insertion(+), 61 deletions(-)
--

[43/50] [abbrv] beam git commit: This closes #3337: Include Additional PTransform Inputs in Transform Nodes

[kenn]

This closes #3337: Include Additional PTransform Inputs in Transform Nodes

  Roll-forward Include Additional PTransform inputs in Transform Nodes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b4c77167
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b4c77167
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b4c77167

Branch: refs/heads/gearpump-runner
Commit: b4c77167f20b8346d22ccff440489592e0de4b52
Parents: 33883ed 696f8b2
Author: Kenneth Knowles 
Authored: Fri Jun 9 15:04:02 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 9 15:04:02 2017 -0700

--
 .../apex/translation/TranslationContext.java|   4 +-
 .../core/construction/TransformInputs.java  |  50 ++
 .../core/construction/TransformInputsTest.java  | 166 +++
 .../apache/beam/runners/direct/DirectGraph.java |  34 +++-
 .../beam/runners/direct/DirectGraphVisitor.java |  26 ++-
 .../direct/ExecutorServiceParallelExecutor.java |   2 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   9 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 +
 .../beam/runners/direct/WatermarkManager.java   |  14 +-
 .../runners/direct/DirectGraphVisitorTest.java  |  10 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   6 +-
 .../flink/FlinkBatchTranslationContext.java |   3 +-
 .../flink/FlinkStreamingTranslationContext.java |   3 +-
 .../dataflow/DataflowPipelineTranslator.java|   5 +-
 .../spark/translation/EvaluationContext.java|   4 +-
 .../beam/sdk/runners/TransformHierarchy.java|  28 +++-
 18 files changed, 323 insertions(+), 46 deletions(-)
--

[29/50] [abbrv] beam git commit: This closes #3220: Implements HCatalogIO

[kenn]

This closes #3220: Implements HCatalogIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/911bfbde
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/911bfbde
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/911bfbde

Branch: refs/heads/gearpump-runner
Commit: 911bfbdef62a7bb750426da9cc59537c4ffd9e1f
Parents: 7c5a70e 349898c
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:07:47 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:07:47 2017 -0700

--
 sdks/java/io/hcatalog/pom.xml   | 163 ++
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 511 +++
 .../beam/sdk/io/hcatalog/package-info.java  |  22 +
 .../io/hcatalog/EmbeddedMetastoreService.java   |  88 
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java| 271 ++
 .../sdk/io/hcatalog/HCatalogIOTestUtils.java| 106 
 .../hcatalog/src/test/resources/hive-site.xml   | 301 +++
 sdks/java/io/pom.xml|   1 +
 8 files changed, 1463 insertions(+)
--

[07/50] [abbrv] beam git commit: This closes #3294

[kenn]

This closes #3294


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/32f22b7d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/32f22b7d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/32f22b7d

Branch: refs/heads/gearpump-runner
Commit: 32f22b7d9cfd5dcd22555272c5a7365fd1323e5f
Parents: caecac3 9c83ffe
Author: Ismaël Mejía 
Authored: Wed Jun 7 23:14:02 2017 +0200
Committer: Ismaël Mejía 
Committed: Wed Jun 7 23:14:02 2017 +0200

--
 .../wrappers/streaming/io/UnboundedSourceWrapper.java   | 5 +
 1 file changed, 5 insertions(+)
--

[23/50] [abbrv] beam git commit: DirectRunner override matchers using Runner API

[kenn]

DirectRunner override matchers using Runner API


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d8d90878
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d8d90878
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d8d90878

Branch: refs/heads/gearpump-runner
Commit: d8d9087877c01f1786271726a541fb3eeda7f939
Parents: ca7b9c2
Author: Kenneth Knowles 
Authored: Thu May 25 06:31:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:36:28 2017 -0700

--
 .../beam/runners/direct/DirectRunner.java   | 20 +---
 1 file changed, 9 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d8d90878/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index dbd1ec4..136ccf3 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -30,6 +30,7 @@ import java.util.Map;
 import java.util.Set;
 import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
 import org.apache.beam.runners.core.construction.SplittableParDo;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory;
@@ -42,12 +43,9 @@ import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.metrics.MetricsEnvironment;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.runners.PTransformOverride;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
-import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.values.PCollection;
 import org.joda.time.Duration;
@@ -230,33 +228,33 @@ public class DirectRunner extends 
PipelineRunner {
 new WriteWithShardingFactory())) /* Uses a view internally. */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(CreatePCollectionView.class),
+
PTransformMatchers.urnEqualTo(PTransformTranslation.CREATE_VIEW_TRANSFORM_URN),
 new ViewOverrideFactory())) /* Uses pardos and GBKs */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(TestStream.class),
+
PTransformMatchers.urnEqualTo(PTransformTranslation.TEST_STREAM_TRANSFORM_URN),
 new DirectTestStreamFactory(this))) /* primitive */
 // SplittableParMultiDo is implemented in terms of nonsplittable 
simple ParDos and extra
 // primitives
 .add(
 PTransformOverride.of(
-PTransformMatchers.splittableParDoMulti(), new 
ParDoMultiOverrideFactory()))
+PTransformMatchers.splittableParDo(), new 
ParDoMultiOverrideFactory()))
 // state and timer pardos are implemented in terms of simple ParDos 
and extra primitives
 .add(
 PTransformOverride.of(
-PTransformMatchers.stateOrTimerParDoMulti(), new 
ParDoMultiOverrideFactory()))
+PTransformMatchers.stateOrTimerParDo(), new 
ParDoMultiOverrideFactory()))
 .add(
 PTransformOverride.of(
-
PTransformMatchers.classEqualTo(SplittableParDo.ProcessKeyedElements.class),
+PTransformMatchers.urnEqualTo(
+SplittableParDo.SPLITTABLE_PROCESS_KEYED_ELEMENTS_URN),
 new SplittableParDoViaKeyedWorkItems.OverrideFactory()))
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(
-
SplittableParDoViaKeyedWorkItems.GBKIntoKeyedWorkItems.class),
+
PTransformMatchers.urnEqualTo(SplittableParDo.SPLITTABLE_GBKIKWI_URN),
 new DirectGBKIntoKeyedWorkItemsOverrideFactory())) /* Returns 
a GBKO */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(GroupByKey.class),
+

[31/50] [abbrv] beam git commit: This closes #3331: Slight debuggability improvements in BigtableIO

[kenn]

This closes #3331: Slight debuggability improvements in BigtableIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bc2f97c7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bc2f97c7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bc2f97c7

Branch: refs/heads/gearpump-runner
Commit: bc2f97c7769bb9b81bc13a181c6e68e5fe219726
Parents: 911bfbd 82a6cb6
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:57:55 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:57:55 2017 -0700

--
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  | 22 
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  2 +-
 2 files changed, 14 insertions(+), 10 deletions(-)
--

[jira] [Commented] (BEAM-2437) quickstart.py docs is missing the path to MANIFEST.in

[Jonathan Bingham (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046749#comment-16046749
 ] 

Jonathan Bingham commented on BEAM-2437:


Works for me!

> quickstart.py docs is missing the path to MANIFEST.in
> -
>
> Key: BEAM-2437
> URL: https://issues.apache.org/jira/browse/BEAM-2437
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Jonathan Bingham
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: easyfix
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> SUMMARY
> The wordcount example in quickstart-py does not work with the sample code 
> without modification.
> OBSERVED
> Copy-pasting from the doc page doesn't work:
> python -m apache_beam.examples.wordcount --input MANIFEST.in --output counts
> Error message: IOError: No files found based on the file pattern MANIFEST.in
> EXPECTED
> The example tells me to set the path to MANIFEST.in, or gives a pseudo-path 
> that I can substitute in the right path prefix.
> python -m apache_beam.examples.wordcount --input 
> /[path-to-git-clone-dir]/beam/sdks/python/MANIFEST.in --output counts



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2437) quickstart.py docs is missing the path to MANIFEST.in

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046742#comment-16046742
 ] 

Ahmet Altay commented on BEAM-2437:
---

Thank you [~j...@jbingham.com] for reporting this. We could change the snippet 
to use {{pom.xml}] as the input file. A file with that name exists both at the 
root and at sdks/python. It would would regardless of where user is running the 
command.

> quickstart.py docs is missing the path to MANIFEST.in
> -
>
> Key: BEAM-2437
> URL: https://issues.apache.org/jira/browse/BEAM-2437
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Jonathan Bingham
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: easyfix
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> SUMMARY
> The wordcount example in quickstart-py does not work with the sample code 
> without modification.
> OBSERVED
> Copy-pasting from the doc page doesn't work:
> python -m apache_beam.examples.wordcount --input MANIFEST.in --output counts
> Error message: IOError: No files found based on the file pattern MANIFEST.in
> EXPECTED
> The example tells me to set the path to MANIFEST.in, or gives a pseudo-path 
> that I can substitute in the right path prefix.
> python -m apache_beam.examples.wordcount --input 
> /[path-to-git-clone-dir]/beam/sdks/python/MANIFEST.in --output counts



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/5] beam git commit: rename SQL to Sql in class name

[lcwik]

http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLFilterFn.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLFilterFn.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLFilterFn.java
deleted file mode 100644
index 2ab6301..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLFilterFn.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.transform;
-
-import java.util.List;
-import org.apache.beam.dsls.sql.interpreter.BeamSQLExpressionExecutor;
-import org.apache.beam.dsls.sql.rel.BeamFilterRel;
-import org.apache.beam.dsls.sql.schema.BeamSQLRow;
-import org.apache.beam.sdk.transforms.DoFn;
-
-/**
- * {@code BeamSQLFilterFn} is the executor for a {@link BeamFilterRel} step.
- *
- */
-public class BeamSQLFilterFn extends DoFn {
-
-  private String stepName;
-  private BeamSQLExpressionExecutor executor;
-
-  public BeamSQLFilterFn(String stepName, BeamSQLExpressionExecutor executor) {
-super();
-this.stepName = stepName;
-this.executor = executor;
-  }
-
-  @Setup
-  public void setup() {
-executor.prepare();
-  }
-
-  @ProcessElement
-  public void processElement(ProcessContext c) {
-BeamSQLRow in = c.element();
-
-List result = executor.execute(in);
-
-if ((Boolean) result.get(0)) {
-  c.output(in);
-}
-  }
-
-  @Teardown
-  public void close() {
-executor.close();
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLOutputToConsoleFn.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLOutputToConsoleFn.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLOutputToConsoleFn.java
deleted file mode 100644
index c146ea5..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLOutputToConsoleFn.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.transform;
-
-import org.apache.beam.dsls.sql.schema.BeamSQLRow;
-import org.apache.beam.sdk.transforms.DoFn;
-
-/**
- * A test PTransform to display output in console.
- *
- */
-public class BeamSQLOutputToConsoleFn extends DoFn {
-
-  private String stepName;
-
-  public BeamSQLOutputToConsoleFn(String stepName) {
-super();
-this.stepName = stepName;
-  }
-
-  @ProcessElement
-  public void processElement(ProcessContext c) {
-System.out.println("Output: " + c.element().getDataValues());
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLProjectFn.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLProjectFn.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLProjectFn.java
deleted file mode 100644
index ef4dc0f..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/transform/BeamSQLProjectFn.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software 

[5/5] beam git commit: [BEAM-2433] unify SQL/Sql words in class name

[lcwik]

[BEAM-2433] unify SQL/Sql words in class name

This closes #3340


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e68badd4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e68badd4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e68badd4

Branch: refs/heads/DSL_SQL
Commit: e68badd4d1d25e10e68900bc07077d2f556ef30d
Parents: 5c1f2cb 743f0b3
Author: Luke Cwik 
Authored: Mon Jun 12 07:48:23 2017 -0700
Committer: Luke Cwik 
Committed: Mon Jun 12 07:48:23 2017 -0700

--
 .../java/org/apache/beam/dsls/sql/BeamSql.java  |  20 +-
 .../org/apache/beam/dsls/sql/BeamSqlCli.java|   8 +-
 .../beam/dsls/sql/example/BeamSqlExample.java   |  16 +-
 .../interpreter/BeamSQLExpressionExecutor.java  |  43 ---
 .../dsls/sql/interpreter/BeamSQLFnExecutor.java | 235 
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 +++
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java | 235 
 .../operator/BeamSqlAndExpression.java  |   4 +-
 .../operator/BeamSqlCaseExpression.java |   4 +-
 .../operator/BeamSqlCompareExpression.java  |   4 +-
 .../interpreter/operator/BeamSqlExpression.java |   8 +-
 .../operator/BeamSqlInputRefExpression.java |   4 +-
 .../operator/BeamSqlIsNotNullExpression.java|   4 +-
 .../operator/BeamSqlIsNullExpression.java   |   4 +-
 .../operator/BeamSqlOrExpression.java   |   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java  |   6 +-
 .../operator/BeamSqlUdfExpression.java  |   4 +-
 .../operator/BeamSqlWindowEndExpression.java|   4 +-
 .../operator/BeamSqlWindowExpression.java   |   4 +-
 .../operator/BeamSqlWindowStartExpression.java  |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java |   6 +-
 .../math/BeamSqlMathUnaryExpression.java|   4 +-
 .../string/BeamSqlCharLengthExpression.java |   4 +-
 .../string/BeamSqlConcatExpression.java |   4 +-
 .../string/BeamSqlInitCapExpression.java|   4 +-
 .../operator/string/BeamSqlLowerExpression.java |   4 +-
 .../string/BeamSqlOverlayExpression.java|   4 +-
 .../string/BeamSqlPositionExpression.java   |   4 +-
 .../string/BeamSqlSubstringExpression.java  |   4 +-
 .../operator/string/BeamSqlTrimExpression.java  |   4 +-
 .../operator/string/BeamSqlUpperExpression.java |   4 +-
 .../beam/dsls/sql/planner/BeamQueryPlanner.java |   4 +-
 .../beam/dsls/sql/planner/BeamSQLRelUtils.java  |  74 
 .../beam/dsls/sql/planner/BeamSqlRelUtils.java  |  74 
 .../beam/dsls/sql/rel/BeamAggregationRel.java   |  58 +--
 .../apache/beam/dsls/sql/rel/BeamFilterRel.java |  28 +-
 .../apache/beam/dsls/sql/rel/BeamIOSinkRel.java |  12 +-
 .../beam/dsls/sql/rel/BeamIOSourceRel.java  |  14 +-
 .../beam/dsls/sql/rel/BeamProjectRel.java   |  28 +-
 .../apache/beam/dsls/sql/rel/BeamRelNode.java   |   4 +-
 .../apache/beam/dsls/sql/rel/BeamSortRel.java   |  32 +-
 .../apache/beam/dsls/sql/rel/BeamValuesRel.java |  16 +-
 .../beam/dsls/sql/schema/BaseBeamTable.java |  10 +-
 .../dsls/sql/schema/BeamPCollectionTable.java   |  10 +-
 .../beam/dsls/sql/schema/BeamSQLRecordType.java |  97 -
 .../apache/beam/dsls/sql/schema/BeamSQLRow.java | 367 ---
 .../beam/dsls/sql/schema/BeamSqlRecordType.java |  96 +
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 367 +++
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   |  16 +-
 .../beam/dsls/sql/schema/BeamTableUtils.java|  10 +-
 .../sql/schema/kafka/BeamKafkaCSVTable.java |  42 +--
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   |  16 +-
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |   6 +-
 .../schema/text/BeamTextCSVTableIOReader.java   |  18 +-
 .../schema/text/BeamTextCSVTableIOWriter.java   |  20 +-
 .../transform/BeamAggregationTransforms.java|  56 +--
 .../dsls/sql/transform/BeamSQLFilterFn.java |  62 
 .../sql/transform/BeamSQLOutputToConsoleFn.java |  41 ---
 .../dsls/sql/transform/BeamSQLProjectFn.java|  72 
 .../dsls/sql/transform/BeamSqlFilterFn.java |  62 
 .../sql/transform/BeamSqlOutputToConsoleFn.java |  41 +++
 .../dsls/sql/transform/BeamSqlProjectFn.java|  72 
 .../beam/dsls/sql/transform/package-info.java   |   2 +-
 .../sql/interpreter/BeamSQLFnExecutorTest.java  | 268 --
 .../interpreter/BeamSQLFnExecutorTestBase.java  |  91 -
 .../sql/interpreter/BeamSqlFnExecutorTest.java  | 268 ++
 .../interpreter/BeamSqlFnExecutorTestBase.java  |  91 +
 .../operator/BeamNullExperssionTest.java|   4 +-
 .../operator/BeamSqlAndOrExpressionTest.java|   4 +-
 .../operator/BeamSqlCaseExpressionTest.java |   4 +-
 .../operator/BeamSqlCompareExpressionTest.java  |   4 +-
 .../operator/BeamSqlInputRefExpressionTest.java |   4 +-
 .../operator/BeamSqlPrimitiveTest.java  |   4 +-
 

[3/5] beam git commit: rename SQL to Sql in class name

[lcwik]

http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamAggregationRel.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamAggregationRel.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamAggregationRel.java
index 6914883..c0d2783 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamAggregationRel.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamAggregationRel.java
@@ -18,9 +18,9 @@
 package org.apache.beam.dsls.sql.rel;
 
 import java.util.List;
-import org.apache.beam.dsls.sql.planner.BeamSQLRelUtils;
-import org.apache.beam.dsls.sql.schema.BeamSQLRecordType;
-import org.apache.beam.dsls.sql.schema.BeamSQLRow;
+import org.apache.beam.dsls.sql.planner.BeamSqlRelUtils;
+import org.apache.beam.dsls.sql.schema.BeamSqlRecordType;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
 import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
 import org.apache.beam.dsls.sql.transform.BeamAggregationTransforms;
 import org.apache.beam.sdk.coders.IterableCoder;
@@ -55,7 +55,7 @@ import org.joda.time.Duration;
  */
 public class BeamAggregationRel extends Aggregate implements BeamRelNode {
   private int windowFieldIdx = -1;
-  private WindowFn windowFn;
+  private WindowFn windowFn;
   private Trigger trigger;
   private Duration allowedLatence = Duration.ZERO;
 
@@ -71,48 +71,48 @@ public class BeamAggregationRel extends Aggregate 
implements BeamRelNode {
   }
 
   @Override
-  public PCollection buildBeamPipeline(PCollectionTuple 
inputPCollections)
+  public PCollection buildBeamPipeline(PCollectionTuple 
inputPCollections)
   throws Exception {
 RelNode input = getInput();
-String stageName = BeamSQLRelUtils.getStageName(this);
+String stageName = BeamSqlRelUtils.getStageName(this);
 
-PCollection upstream =
-
BeamSQLRelUtils.getBeamRelInput(input).buildBeamPipeline(inputPCollections);
+PCollection upstream =
+
BeamSqlRelUtils.getBeamRelInput(input).buildBeamPipeline(inputPCollections);
 if (windowFieldIdx != -1) {
   upstream = upstream.apply("assignEventTimestamp", WithTimestamps
-  .of(new 
BeamAggregationTransforms.WindowTimestampFn(windowFieldIdx)))
+  .of(new 
BeamAggregationTransforms.WindowTimestampFn(windowFieldIdx)))
   .setCoder(upstream.getCoder());
 }
 
-PCollection windowStream = upstream.apply("window",
-Window.into(windowFn)
+PCollection windowStream = upstream.apply("window",
+Window.into(windowFn)
 .triggering(trigger)
 .withAllowedLateness(allowedLatence)
 .accumulatingFiredPanes());
 
 BeamSqlRowCoder keyCoder = new 
BeamSqlRowCoder(exKeyFieldsSchema(input.getRowType()));
-PCollection> exGroupByStream = 
windowStream.apply("exGroupBy",
+PCollection> exGroupByStream = 
windowStream.apply("exGroupBy",
 WithKeys
 .of(new BeamAggregationTransforms.AggregationGroupByKeyFn(
 windowFieldIdx, groupSet)))
-.setCoder(KvCoder.of(keyCoder, 
upstream.getCoder()));
+.setCoder(KvCoder.of(keyCoder, 
upstream.getCoder()));
 
-PCollection> groupedStream = 
exGroupByStream
-.apply("groupBy", GroupByKey.create())
-.setCoder(KvCoder.of(keyCoder,
-IterableCoder.of(upstream.getCoder(;
+PCollection> groupedStream = 
exGroupByStream
+.apply("groupBy", GroupByKey.create())
+.setCoder(KvCoder.of(keyCoder,
+IterableCoder.of(upstream.getCoder(;
 
 BeamSqlRowCoder aggCoder = new BeamSqlRowCoder(exAggFieldsSchema());
-PCollection> aggregatedStream = 
groupedStream.apply("aggregation",
-Combine.groupedValues(
+PCollection> aggregatedStream = 
groupedStream.apply("aggregation",
+Combine.groupedValues(
 new 
BeamAggregationTransforms.AggregationCombineFn(getAggCallList(),
-BeamSQLRecordType.from(input.getRowType()
-.setCoder(KvCoder.of(keyCoder, aggCoder));
+BeamSqlRecordType.from(input.getRowType()
+.setCoder(KvCoder.of(keyCoder, aggCoder));
 
-PCollection mergedStream = 
aggregatedStream.apply("mergeRecord",
+PCollection mergedStream = 
aggregatedStream.apply("mergeRecord",
 ParDo.of(new BeamAggregationTransforms.MergeAggregationRecord(
-BeamSQLRecordType.from(getRowType()), 

[1/5] beam git commit: rename SQL to Sql in class name

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL 5c1f2cbc6 -> e68badd4d


http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/BeamPlannerSubmitTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/BeamPlannerSubmitTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/BeamPlannerSubmitTest.java
index 8a48618..4df7f8a 100644
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/BeamPlannerSubmitTest.java
+++ 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/BeamPlannerSubmitTest.java
@@ -18,7 +18,7 @@
 package org.apache.beam.dsls.sql.planner;
 
 import org.apache.beam.dsls.sql.BeamSqlCli;
-import org.apache.beam.dsls.sql.schema.BeamSQLRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.values.PCollection;
 import org.junit.Assert;
@@ -36,7 +36,7 @@ public class BeamPlannerSubmitTest extends BasePlanner {
 
   @Before
   public void prepare() {
-MockedBeamSQLTable.CONTENT.clear();
+MockedBeamSqlTable.CONTENT.clear();
   }
 
   @Test
@@ -45,12 +45,12 @@ public class BeamPlannerSubmitTest extends BasePlanner {
 + " order_id, site_id, price "
 + "FROM ORDER_DETAILS " + "WHERE SITE_ID = 0 and price > 20";
 
-PCollection outputStream = BeamSqlCli.compilePipeline(sql, 
pipeline);
+PCollection outputStream = BeamSqlCli.compilePipeline(sql, 
pipeline);
 
 pipeline.run().waitUntilFinish();
 
-Assert.assertTrue(MockedBeamSQLTable.CONTENT.size() == 1);
-Assert.assertTrue(MockedBeamSQLTable.CONTENT.peek().valueInString()
+Assert.assertTrue(MockedBeamSqlTable.CONTENT.size() == 1);
+Assert.assertTrue(MockedBeamSqlTable.CONTENT.peek().valueInString()
 .contains("order_id=12345,site_id=0,price=20.5,order_time="));
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/743f0b3b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/MockedBeamSQLTable.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/MockedBeamSQLTable.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/MockedBeamSQLTable.java
deleted file mode 100644
index 561f4be..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/planner/MockedBeamSQLTable.java
+++ /dev/null
@@ -1,161 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.planner;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.ConcurrentLinkedQueue;
-
-import org.apache.beam.dsls.sql.schema.BaseBeamTable;
-import org.apache.beam.dsls.sql.schema.BeamIOType;
-import org.apache.beam.dsls.sql.schema.BeamSQLRecordType;
-import org.apache.beam.dsls.sql.schema.BeamSQLRow;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PDone;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelProtoDataType;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * A mock table use to check input/output.
- *
- */
-public class MockedBeamSQLTable extends BaseBeamTable {
-
-  public static final ConcurrentLinkedQueue CONTENT = new 
ConcurrentLinkedQueue<>();
-
-  private List inputRecords;
-
-  public MockedBeamSQLTable(RelProtoDataType protoRowType) {
-super(protoRowType);
-  }
-
-  public MockedBeamSQLTable withInputRecords(List inputRecords){
-this.inputRecords = inputRecords;
-return this;
-  }
-
-  /**
-   * Convenient way to build a mocked table with mock data:
-   *
-   * e.g.
-   *
-   * {@code
-   * MockedBeamSQLTable
-   *   .of(SqlTypeName.BIGINT, "order_id",
-   *   SqlTypeName.INTEGER, "site_id",
-   *   

[4/5] beam git commit: rename SQL to Sql in class name

[lcwik]

rename SQL to Sql in class name


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/743f0b3b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/743f0b3b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/743f0b3b

Branch: refs/heads/DSL_SQL
Commit: 743f0b3b62616d1c8fe77d306920fbcce81c8855
Parents: 5c1f2cb
Author: mingmxu 
Authored: Fri Jun 9 23:07:00 2017 -0700
Committer: Luke Cwik 
Committed: Mon Jun 12 07:47:40 2017 -0700

--
 .../java/org/apache/beam/dsls/sql/BeamSql.java  |  20 +-
 .../org/apache/beam/dsls/sql/BeamSqlCli.java|   8 +-
 .../beam/dsls/sql/example/BeamSqlExample.java   |  16 +-
 .../interpreter/BeamSQLExpressionExecutor.java  |  43 ---
 .../dsls/sql/interpreter/BeamSQLFnExecutor.java | 235 
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 +++
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java | 235 
 .../operator/BeamSqlAndExpression.java  |   4 +-
 .../operator/BeamSqlCaseExpression.java |   4 +-
 .../operator/BeamSqlCompareExpression.java  |   4 +-
 .../interpreter/operator/BeamSqlExpression.java |   8 +-
 .../operator/BeamSqlInputRefExpression.java |   4 +-
 .../operator/BeamSqlIsNotNullExpression.java|   4 +-
 .../operator/BeamSqlIsNullExpression.java   |   4 +-
 .../operator/BeamSqlOrExpression.java   |   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java  |   6 +-
 .../operator/BeamSqlUdfExpression.java  |   4 +-
 .../operator/BeamSqlWindowEndExpression.java|   4 +-
 .../operator/BeamSqlWindowExpression.java   |   4 +-
 .../operator/BeamSqlWindowStartExpression.java  |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java |   6 +-
 .../math/BeamSqlMathUnaryExpression.java|   4 +-
 .../string/BeamSqlCharLengthExpression.java |   4 +-
 .../string/BeamSqlConcatExpression.java |   4 +-
 .../string/BeamSqlInitCapExpression.java|   4 +-
 .../operator/string/BeamSqlLowerExpression.java |   4 +-
 .../string/BeamSqlOverlayExpression.java|   4 +-
 .../string/BeamSqlPositionExpression.java   |   4 +-
 .../string/BeamSqlSubstringExpression.java  |   4 +-
 .../operator/string/BeamSqlTrimExpression.java  |   4 +-
 .../operator/string/BeamSqlUpperExpression.java |   4 +-
 .../beam/dsls/sql/planner/BeamQueryPlanner.java |   4 +-
 .../beam/dsls/sql/planner/BeamSQLRelUtils.java  |  74 
 .../beam/dsls/sql/planner/BeamSqlRelUtils.java  |  74 
 .../beam/dsls/sql/rel/BeamAggregationRel.java   |  58 +--
 .../apache/beam/dsls/sql/rel/BeamFilterRel.java |  28 +-
 .../apache/beam/dsls/sql/rel/BeamIOSinkRel.java |  12 +-
 .../beam/dsls/sql/rel/BeamIOSourceRel.java  |  14 +-
 .../beam/dsls/sql/rel/BeamProjectRel.java   |  28 +-
 .../apache/beam/dsls/sql/rel/BeamRelNode.java   |   4 +-
 .../apache/beam/dsls/sql/rel/BeamSortRel.java   |  32 +-
 .../apache/beam/dsls/sql/rel/BeamValuesRel.java |  16 +-
 .../beam/dsls/sql/schema/BaseBeamTable.java |  10 +-
 .../dsls/sql/schema/BeamPCollectionTable.java   |  10 +-
 .../beam/dsls/sql/schema/BeamSQLRecordType.java |  97 -
 .../apache/beam/dsls/sql/schema/BeamSQLRow.java | 367 ---
 .../beam/dsls/sql/schema/BeamSqlRecordType.java |  96 +
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 367 +++
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   |  16 +-
 .../beam/dsls/sql/schema/BeamTableUtils.java|  10 +-
 .../sql/schema/kafka/BeamKafkaCSVTable.java |  42 +--
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   |  16 +-
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |   6 +-
 .../schema/text/BeamTextCSVTableIOReader.java   |  18 +-
 .../schema/text/BeamTextCSVTableIOWriter.java   |  20 +-
 .../transform/BeamAggregationTransforms.java|  56 +--
 .../dsls/sql/transform/BeamSQLFilterFn.java |  62 
 .../sql/transform/BeamSQLOutputToConsoleFn.java |  41 ---
 .../dsls/sql/transform/BeamSQLProjectFn.java|  72 
 .../dsls/sql/transform/BeamSqlFilterFn.java |  62 
 .../sql/transform/BeamSqlOutputToConsoleFn.java |  41 +++
 .../dsls/sql/transform/BeamSqlProjectFn.java|  72 
 .../beam/dsls/sql/transform/package-info.java   |   2 +-
 .../sql/interpreter/BeamSQLFnExecutorTest.java  | 268 --
 .../interpreter/BeamSQLFnExecutorTestBase.java  |  91 -
 .../sql/interpreter/BeamSqlFnExecutorTest.java  | 268 ++
 .../interpreter/BeamSqlFnExecutorTestBase.java  |  91 +
 .../operator/BeamNullExperssionTest.java|   4 +-
 .../operator/BeamSqlAndOrExpressionTest.java|   4 +-
 .../operator/BeamSqlCaseExpressionTest.java |   4 +-
 .../operator/BeamSqlCompareExpressionTest.java  |   4 +-
 .../operator/BeamSqlInputRefExpressionTest.java |   4 +-
 .../operator/BeamSqlPrimitiveTest.java  |   4 +-
 .../operator/BeamSqlUdfExpressionTest.java  

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2337

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2409) Spark runner produces exactly twice the number of results in streaming mode when use triggers to re-window results on global window.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-2409:
---

Assignee: Amit Sela  (was: Aviem Zur)

> Spark runner produces exactly twice the number of results in streaming mode 
> when use triggers to re-window results on global window.
> 
>
> Key: BEAM-2409
> URL: https://issues.apache.org/jira/browse/BEAM-2409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.0.0
>Reporter: Ismaël Mejía
>Assignee: Amit Sela
>
> This can be tested with Nexmark query 6. Sorry I don’t have a smaller test 
> case than this, but I think the part of the pipeline that produces the result 
> is this one.
> {code:java}
> .apply(
> Window.>into(new GlobalWindows())
> 
> .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
> .accumulatingFiredPanes()
> .withAllowedLateness(Duration.ZERO))
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3343: [BEAM-2359] Fix watermark broadcasting to executors...

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/3343

[BEAM-2359] Fix watermark broadcasting to executors in Spark runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @amitsela 
CC: @staslev @kobisalant 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam BEAM-2359-watermark-bug-spark

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3343.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3343


commit f1b679e402734f20dcd9645babaec0a3f291e259
Author: Aviem Zur 
Date:   2017-06-12T14:04:00Z

[BEAM-2359] Fix watermark broadcasting to executors in Spark runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-2429) Conflicting filesystems with used of HadoopFileSystem

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

François Wagner resolved BEAM-2429.
---
   Resolution: Not A Bug
Fix Version/s: 2.0.0

> Conflicting filesystems with used of HadoopFileSystem
> -
>
> Key: BEAM-2429
> URL: https://issues.apache.org/jira/browse/BEAM-2429
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: François Wagner
>Assignee: Flavio Fiszman
> Fix For: 2.0.0
>
>
> I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks 
> like HadoopFileSystem is registring itself under the `file` schema 
> (https://github.com/apache/beam/pull/2777/files#diff-330bd0854dcab6037ef0e52c05d68eb2L79),
>  hence the following Exception is thrown when trying to register 
> HadoopFileSystem.
> java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems: 
> [org.apache.beam.sdk.io.LocalFileSystem, 
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem]
>   at 
> org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:498)
> What is the correct way to handle `hdfs` url out of the box with TextIO & 
> AvroIO ?
> {code:java}
> String[] args = new String[]{
> "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\": 
> \"true\"}]"};
> HadoopFileSystemOptions options = PipelineOptionsFactory
> .fromArgs(args)
> .withValidation()
> .as(HadoopFileSystemOptions.class);
> Pipeline pipeline = Pipeline.create(options); 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2429) Conflicting filesystems with used of HadoopFileSystem

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046511#comment-16046511
 ] 

François Wagner commented on BEAM-2429:
---

Hi Flavio, Thanks for your input, it worked right away when I've added 
"fs.defaultFS". Maybe one could add this somewhere in the documentation as it's 
not obvious that we have to add this option to handle "hdfs://" URI, moreover 
that was not the case with the previous version of HdfsIO. Thanks a lot for 
your help. François

> Conflicting filesystems with used of HadoopFileSystem
> -
>
> Key: BEAM-2429
> URL: https://issues.apache.org/jira/browse/BEAM-2429
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: François Wagner
>Assignee: Flavio Fiszman
>
> I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks 
> like HadoopFileSystem is registring itself under the `file` schema 
> (https://github.com/apache/beam/pull/2777/files#diff-330bd0854dcab6037ef0e52c05d68eb2L79),
>  hence the following Exception is thrown when trying to register 
> HadoopFileSystem.
> java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems: 
> [org.apache.beam.sdk.io.LocalFileSystem, 
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem]
>   at 
> org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:498)
> What is the correct way to handle `hdfs` url out of the box with TextIO & 
> AvroIO ?
> {code:java}
> String[] args = new String[]{
> "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\": 
> \"true\"}]"};
> HadoopFileSystemOptions options = PipelineOptionsFactory
> .fromArgs(args)
> .withValidation()
> .as(HadoopFileSystemOptions.class);
> Pipeline pipeline = Pipeline.create(options); 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #124

[Apache Jenkins Server]

See 


--
[...truncated 2.64 MB...]
2017-06-12T12:35:01.971 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-12T12:35:02.009 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 296.2 KB/sec)
2017-06-12T12:35:02.014 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-12T12:35:02.024 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1283.4 KB/sec)
2017-06-12T12:35:02.029 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-12T12:35:02.039 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 217.3 KB/sec)
2017-06-12T12:35:02.043 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-12T12:35:02.087 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 35.8 KB/sec)
2017-06-12T12:35:02.092 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-12T12:35:02.102 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 525.0 KB/sec)
2017-06-12T12:35:02.106 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-12T12:35:02.115 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 277.2 KB/sec)
2017-06-12T12:35:02.121 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-12T12:35:02.144 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 149.0 KB/sec)
2017-06-12T12:35:02.151 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-12T12:35:02.161 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 100.3 KB/sec)
2017-06-12T12:35:02.167 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-12T12:35:02.177 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 188.5 KB/sec)
2017-06-12T12:35:02.183 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-12T12:35:02.193 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 176.4 KB/sec)
2017-06-12T12:35:02.199 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-12T12:35:02.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 241.3 KB/sec)
2017-06-12T12:35:02.215 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-12T12:35:02.228 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1355.1 KB/sec)
2017-06-12T12:35:02.233 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-12T12:35:02.242 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 1047.5 KB/sec)
2017-06-12T12:35:02.246 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-12T12:35:02.258 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 404.9 KB/sec)
2017-06-12T12:35:02.263 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-12T12:35:02.272 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
 (3 KB at 256.2 KB/sec)
2017-06-12T12:35:02.276 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
2017-06-12T12:35:02.288 [INFO] Downloaded: 

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #76

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #76

[Apache Jenkins Server]

See 


--
[...truncated 1.56 MB...]
  [javadoc]   ^
  [javadoc]   symbol:   class CompletableFuture
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:66:
 error: cannot find symbol
  [javadoc]   Supplier processBundleInstructionIdSupplier,
  [javadoc]   ^
  [javadoc]   symbol:   class Supplier
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:51:
 warning - Tag @link: reference not found: DoFn.FinishBundle @FinishBundle
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:51:
 warning - Tag @link: reference not found: DoFn.FinishBundle @FinishBundle
  [javadoc] 
:34:
 warning - Tag @link: reference not found: Source.Reader
  [javadoc] 
:292:
 warning - Tag @link: reference not found: UnboundedSource.CheckpointMark
  [javadoc] 
:101:
 warning - Tag @link: reference not found: UnboundedSource.CheckpointMark
  [javadoc] 
:101:
 warning - Tag @link: reference not found: UnboundedSource.CheckpointMark
  [javadoc]