[jira] [Commented] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309315#comment-15309315
 ] 

ASF GitHub Bot commented on BEAM-242:
-

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/382


> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-242] Fix javadoc on Flink runner.

[jbonofre]

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/382


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-261) Apache Apex Runner

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309223#comment-15309223
 ] 

Thomas Weise commented on BEAM-261:
---

Working on Apex runner PoC, but cannot assign this JIRA to myself.



> Apache Apex Runner
> --
>
> Key: BEAM-261
> URL: https://issues.apache.org/jira/browse/BEAM-261
> Project: Beam
>  Issue Type: Wish
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>
> Like Spark, Flink and GearPump, Apache Apex also does have advantages. Is it 
> possible to have a runner for Apache Apex?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #453

[Apache Jenkins Server]

See 

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/0.1.0-incubating-1 [created] a5a289b6c

incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of 0.1.0-incubating-1

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.1.0-incubating 2a25d32f0 -> 962d8da06


[maven-release-plugin] rollback changes from release preparation of 
0.1.0-incubating-1


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/962d8da0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/962d8da0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/962d8da0

Branch: refs/heads/release-0.1.0-incubating
Commit: 962d8da06c890d90d33e31c01335f28fd5a2ef31
Parents: 2a25d32
Author: Davor Bonaci 
Authored: Tue May 31 19:48:53 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 19:48:53 2016 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/examples/pom.xml  | 2 +-
 runners/flink/pom.xml   | 4 ++--
 runners/flink/runner/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hdfs/pom.xml   | 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 26 files changed, 28 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 66d7564..351e0a1 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 parent
-0.1.0-incubating
+0.1.0-incubating-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 1e42382..c3073b3 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 parent
-0.1.0-incubating
+0.1.0-incubating-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 15344f9..3657564 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 parent
-0.1.0-incubating
+0.1.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/pom.xml
--
diff --git a/pom.xml b/pom.xml
index f1046e7..757383e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.1.0-incubating
+  0.1.0-incubating-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-0.1.0-incubating-1
+HEAD
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index 042286f..2a8f976 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam.runners
 parent
-0.1.0-incubating
+0.1.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/962d8da0/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 2afa3e0..0c4e994 100644
--- a/runners/direct-java/pom.xml
+

incubator-beam git commit: [maven-release-plugin] prepare release 0.1.0-incubating-1

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.1.0-incubating [created] 2a25d32f0


[maven-release-plugin] prepare release 0.1.0-incubating-1


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a25d32f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a25d32f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a25d32f

Branch: refs/heads/release-0.1.0-incubating
Commit: 2a25d32f0b1de4ca8473f42473fd4419306cff90
Parents: a829d9c
Author: Davor Bonaci 
Authored: Tue May 31 19:48:38 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 19:48:38 2016 -0700

--
 examples/java/pom.xml   | 6 ++
 examples/java8/pom.xml  | 6 ++
 examples/pom.xml| 6 ++
 pom.xml | 9 -
 runners/core-java/pom.xml   | 6 ++
 runners/direct-java/pom.xml | 6 ++
 runners/flink/examples/pom.xml  | 8 +++-
 runners/flink/pom.xml   | 8 +++-
 runners/flink/runner/pom.xml| 6 ++
 runners/google-cloud-dataflow-java/pom.xml  | 6 ++
 runners/pom.xml | 6 ++
 runners/spark/pom.xml   | 9 +++--
 sdks/java/build-tools/pom.xml   | 6 ++
 sdks/java/core/pom.xml  | 6 ++
 sdks/java/extensions/join-library/pom.xml   | 6 ++
 sdks/java/extensions/pom.xml| 6 ++
 sdks/java/io/google-cloud-platform/pom.xml  | 6 ++
 sdks/java/io/hdfs/pom.xml   | 6 ++
 sdks/java/io/kafka/pom.xml  | 6 ++
 sdks/java/io/pom.xml| 6 ++
 sdks/java/java8tests/pom.xml| 6 ++
 sdks/java/maven-archetypes/examples/pom.xml | 6 ++
 sdks/java/maven-archetypes/pom.xml  | 6 ++
 sdks/java/maven-archetypes/starter/pom.xml  | 6 ++
 sdks/java/pom.xml   | 6 ++
 sdks/pom.xml| 6 ++
 26 files changed, 57 insertions(+), 109 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2a25d32f/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index eb77c88..66d7564 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -15,16 +15,14 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
 
   4.0.0
 
   
 org.apache.beam
 parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2a25d32f/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index b082092..1e42382 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -15,16 +15,14 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
 
   4.0.0
 
   
 org.apache.beam
 parent
-0.1.0-incubating-SNAPSHOT
+0.1.0-incubating
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2a25d32f/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 26b6720..15344f9 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -15,16 +15,14 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+http://maven.apache.org/POM/4.0.0"; 
xmln

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #452

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: This closes #403

[davor]

This closes #403


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a829d9ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a829d9ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a829d9ca

Branch: refs/heads/master
Commit: a829d9caff8011d9d2c0beb588fd249392282dfc
Parents: b2f6950 f5e307e
Author: Davor Bonaci 
Authored: Tue May 31 18:54:40 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 18:54:40 2016 -0700

--
 examples/java8/pom.xml | 4 +---
 runners/flink/examples/pom.xml | 4 +---
 runners/flink/runner/pom.xml   | 4 +---
 3 files changed, 3 insertions(+), 9 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-242] Set javadoc less strict around missing return or param statements

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b2f69504a -> a829d9caf


[BEAM-242] Set javadoc less strict around missing return or param statements


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5e307ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5e307ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5e307ef

Branch: refs/heads/master
Commit: f5e307efa8c8d35e9eea46063c16d8f3044634b5
Parents: b2f6950
Author: Jean-Baptiste Onofré 
Authored: Tue May 31 21:46:46 2016 +0200
Committer: Davor Bonaci 
Committed: Tue May 31 18:54:21 2016 -0700

--
 examples/java8/pom.xml | 4 +---
 runners/flink/examples/pom.xml | 4 +---
 runners/flink/runner/pom.xml   | 4 +---
 3 files changed, 3 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5e307ef/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 8727f8d..b082092 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -39,13 +39,11 @@
 
   
 
-  
   
 org.apache.maven.plugins
 maven-javadoc-plugin
 
-  true
+  -Xdoclint:missing
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5e307ef/runners/flink/examples/pom.xml
--
diff --git a/runners/flink/examples/pom.xml b/runners/flink/examples/pom.xml
index 47e0167..8dff21b 100644
--- a/runners/flink/examples/pom.xml
+++ b/runners/flink/examples/pom.xml
@@ -92,13 +92,11 @@
   
   -->
 
-  
   
 org.apache.maven.plugins
 maven-javadoc-plugin
 
-  true
+  -Xdoclint:missing
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5e307ef/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 82104fd..6b628f1 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -151,13 +151,11 @@
   
   -->
 
-  
   
 org.apache.maven.plugins
 maven-javadoc-plugin
 
-  true
+  -Xdoclint:missing
 
   
 

[GitHub] incubator-beam pull request: [BEAM-242] Set javadoc less strict around missi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/403


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309061#comment-15309061
 ] 

ASF GitHub Bot commented on BEAM-242:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/403


> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-318) AvroCoder is vulnerable to AVRO-607

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308972#comment-15308972
 ] 

ASF GitHub Bot commented on BEAM-318:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/408

[BEAM-318] AvroCoder,Source: workaround AVRO-607

See https://issues.apache.org/jira/browse/AVRO-607

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam avro-sync-workaround

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #408


commit 1673b99ecedbc7c78dcb15d5714786854c9748b1
Author: Dan Halperin 
Date:   2016-06-01T00:42:06Z

AvroCoder,Source: workaround AVRO-607

See https://issues.apache.org/jira/browse/AVRO-607




> AvroCoder is vulnerable to AVRO-607
> ---
>
> Key: BEAM-318
> URL: https://issues.apache.org/jira/browse/BEAM-318
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> See AVRO-607



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-318) AvroCoder is vulnerable to AVRO-607

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-318:


 Summary: AvroCoder is vulnerable to AVRO-607
 Key: BEAM-318
 URL: https://issues.apache.org/jira/browse/BEAM-318
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Daniel Halperin
Assignee: Daniel Halperin


See AVRO-607





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: Add KEYS file

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/405


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add KEYS file

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 00ab9b5b5 -> b2f69504a


Add KEYS file


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/65753190
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/65753190
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/65753190

Branch: refs/heads/master
Commit: 65753190edbdbdb0740308c1029b69c7b7116620
Parents: 00ab9b5
Author: Jean-Baptiste Onofré 
Authored: Tue May 31 22:07:48 2016 +0200
Committer: Davor Bonaci 
Committed: Tue May 31 17:42:54 2016 -0700

--
 KEYS | 141 ++
 1 file changed, 141 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/65753190/KEYS
--
diff --git a/KEYS b/KEYS
new file mode 100644
index 000..1f14625
--- /dev/null
+++ b/KEYS
@@ -0,0 +1,141 @@
+This file contains the PGP keys of various developers.
+
+Users: pgp < KEYS
+   gpg --import KEYS
+Developers: 
+pgp -kxa  and append it to this file.
+(pgpk -ll  && pgpk -xa ) >> this file.
+(gpg --list-sigs 
+ && gpg --armor --export ) >> this file.
+ 
+
+pub   4096R/C8282E76 2009-09-08
+uid  Jean-Baptiste Onofré 
+sig 3C8282E76 2009-09-08  Jean-Baptiste Onofré 
+sub   4096R/9F043BBC 2009-09-08
+sig  C8282E76 2009-09-08  Jean-Baptiste Onofré 
+
+-BEGIN PGP PUBLIC KEY BLOCK-
+Version: GnuPG v1
+
+mQINBEqmJkEBEADAAMOjOidXzoyK4FK9WhhRg2EEGX1gm5lK8PpJtk68Fqmz6xvv
+N8VJXMIJUgeD7M35zZSQUWJY43xEU8Yfn6oLL0KR0dIqVOclxE+7G8vxXFcIbRE9
+ziZFp7Z5yzsdzjiIzXv5MVQMczcAAMev/i0BnjiRy5Cg+k6kHXVpu/Gsn05JKPaG
+s7ZcfSxpboyS99MVKQvoFLE5Z/Shh4gFJn2rFInqK5EgVpoZbVyysF52nx0dti/e
+O0NjraQkrEDBWvsPt3cYZA0oP1gWiZiRvOLfAFIarf3poMDyoWBIwnbqb3Msv09j
+yDAmcGq9wsD3alHFHcRIiJl5SzFUStml1d5x/BvUl/Xc5VfHPi2ObKF3xOPGkyTf
+aZ6mYFLaRCAJ0v2MPW+4/grDXKsP8n8xPbE2VQvHBpxaZklD7q4Omn2d+m2sUOLX
+NRUo4n29NyfowAffBYl7ZqrYBBodR9YngWC9LpgM+APHyiw3HzauZ94bGy5Of3+L
+Yu6/riDcP4OXF6r6IH6KIsVqIkv5xzq7OGxxXmlhWg8ifNPLq5yNRccS0nWXc5BD
+/9q06ta/ceQGNkXL327XPuZC+lstWGAa4dKEosRDgcO0Pv2j2a3h8W8oHyxF+gEe
+O+9s0mGdQFxNiEA+JyeKCg+jvfx9Hv/2Syrlert76NEkfbaTFA7BJ4c3EQARAQAB
+tCtKZWFuLUJhcHRpc3RlIE9ub2Zyw6kgPGpib25vZnJlQGFwYWNoZS5vcmc+iQI2
+BBMBAgAgBQJKpiZBAhsDBgsJCAcDAgQVAggDBBYCAwECHgECF4AACgkQv/LuQsgo
+LnboyRAAguqFIpiKkCCR6TR0Y5UQDFhgEMhBreQKCEW0czbGoFnxfULV9H1kJRSB
+Vt0knecGaYS340WEmz4B7BMpkBCgaszgn66+fhacZTBd+Aff1k2lbhdMgdBvlPcm
+q9vFGtbE515j9bPHzsPRJ2wFWd6ot9wXiLD3RJLV6c7L3Egstu3qTp0tEoFHrQps
+qskGBl+mahhMyz3BUDlusavB0Y0tb6hhXCR79ErhjQrTgU947isztYWpgJlA40lx
+DW0hskZWbuGNXjxUJvTT3pKiYUN32WG+2CDNYHceuhsfRLxO/Wb4BKwwDaHWAlH9
+d5F9/vhdPObSv5GQbuUtmCEzeqADUd65jLLM7WSlvRJ+i4m0/TTeP8y4NfxlVbBP
+WuYrQW4gPmDKEDNvEec6PH6hhBfMLJz3M6o4huwLp2kQrq6wSTMDGIoxOLP0ae3c
+BMIuFM5EavLDJmuATUIWWyZt/c7mmAOOh5TGcFWTugnJ6l4FllOrFPiWyFsjMn+U
+zzzaeSkYmq/xZYxjRTdWjK5Zb5rbVuCx/q5VF9Awdy4EM6UXhaqWo06VyjWNOJ86
+wgres4+bVldB7+TiVi9iO6n80WNlPgIaQJlLc+FRsld4Er21kdXreX5doxFD5Iue
+S4y/pLwftHfx1xxj+p2jPJ49Hb0ddNr+XrsrO5txing2pNJgfH65Ag0ESqYmQQEQ
+AKPoXgIIKnyJiPvks7xBV+FqJPecVAx3SSlLyTfsh/jBat9QLd4hsfiZcv1ANZHB
+n4qDeGlsmJ6uDGv8wnUZQ2Im8Heje1h7dKeLNpNnxfBS9gn6e2bXKhAsJGUE7gip
+qVfijFnEY0Vj6Tztzq+Wyqg2Gbz+bJZMo1JVQiaAYyQeQlrOcoZcQHsA/Ol+y48h
+Le36A1TSIPMOSI4ZAZXkqxXAumEaMaz82EvV8KDH7Ijr23Y0wZjEUJ+dJQM9ssuE
+f9GMLIuCbmM/CJ5MCCwepGJd52ymllvgJTHC7B+BY/jKNMWHwAsMJ1oWcPlLzFQI
+Bmyy5RjKoMifzaoSo/hTWkiwcL2Vc+qU3b3/2eUtnCnBB/nkrZkJNNc+OV5YGBSP
+vNPaN43Gvjbvborv4PBvt7QhVjZYQemtXO2sWx1XWSFsucD2K4kJ8ipNWxVgIqDu
+J8SJOnGigX9hMpsZ2HVAwOeKP/jI90J3voKrCPLaKcL1Ip+b28k0aj7kl44YJqw4
+5pbRSx/v73bH4uleQiXSW+JczA+KLw7hX3tOWJEnLS2+Ig9sNUKYGZOg0nw613bN
+fZy8Cbx/UkT10Lznx9FW6MedGyJPYT4MJMMh/PnnsWv50jFnfu2rtnRXEOUXwujL
+fwrmCYbXHgE3Ka+fmRz8HxsyTmtqIHtPixw8RoqfoFfxABEBAAGJAh8EGAECAAkF
+AkqmJkECGwwACgkQv/LuQsgoLnb8AQ/+POsLFdqNqSKfwBXp1YOIEjNdbVjysQc6
+zC6LlMJXNSxAmUmol2g9bJYh9LdpvOTU3gfFgIanaGytC75U7/NOl0zEsN4IU18j
+CLBNaD5/Or1ciQ3CVrID/lPO8s0Hm0/cUPreEjJPPrrPbXG+i9bweg3Dtfy3+WQl
+PhfpvgudwtUjB3st2gztYipkUhmrH+STbbJZVJN5ZNL8mOoM5M2wGS+9VweOWbKe
+z0QeZ9hIPyQNMzTn1xlvRUVNTu8fz2FGvumrd+zgzYcpTE5VpFkOxxUayr3aWXSf
+Cak+HH0WjUDWc9/lJR4dVpwdjLonJfiC70W07J4CnNodYwnPUaGKTVYq3pvQzAPw
+hjx4u6t5zTZy5CbCAEhZC/9GeQmtuM0rcQhz048don4s4baDrqUPKL+X3C3ev4/o
+00yLrQ5rLX8K4iE/Go5xUyhzT7gqFJUPWdo8neTXXwQGThqqhVQovnn3M6i55rCg
+EeOTd7uW+k3vt6kunWZFKPjzRBxMD4NYovIQXwhPxj0vq6DnE0RQa7Dfm6l3cAV7
+/l3kRQcT69AWXotUJQnpY4bemTuYlxAYWCkTGNLdNNiBhiaqlR7xgYMNXS4XqcgA
+6QtP8ulb2FPR0MWEtvGkbHgAAIayV+Jt1Ed2JkIsdJHGeSZO5WEiupySDQCGn6rZ
+DR2E2zua3tQ=
+=LCyH
+-END PGP PUBLIC KEY BLOCK-
+pub   2048R/8F0D334F 2014-10-31
+uid  Davor Bonaci 
+sig 38F0D334F 2014-10-31  Davor Bonaci 
+sub   2048R/D1B59DCE 2014-10-31
+sig  

[2/2] incubator-beam git commit: This closes #405

[davor]

This closes #405


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2f69504
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2f69504
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2f69504

Branch: refs/heads/master
Commit: b2f69504a6fadddac2c7fa0fc4f7438e5288e197
Parents: 00ab9b5 6575319
Author: Davor Bonaci 
Authored: Tue May 31 17:43:06 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 17:43:06 2016 -0700

--
 KEYS | 141 ++
 1 file changed, 141 insertions(+)
--

[GitHub] incubator-beam pull request: [BEAM-318] AvroCoder,Source: workaround AVRO-60...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/408

[BEAM-318] AvroCoder,Source: workaround AVRO-607

See https://issues.apache.org/jira/browse/AVRO-607

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam avro-sync-workaround

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #408


commit 1673b99ecedbc7c78dcb15d5714786854c9748b1
Author: Dan Halperin 
Date:   2016-06-01T00:42:06Z

AvroCoder,Source: workaround AVRO-607

See https://issues.apache.org/jira/browse/AVRO-607




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308901#comment-15308901
 ] 

Pawel Szczur commented on BEAM-315:
---

I've found one related bug in Flink itself:
https://issues.apache.org/jira/browse/FLINK-3291

I've grepped a Beam source for this {{ExecutionConfig.enableObjectReuse}}, no 
matches.

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308890#comment-15308890
 ] 

Pawel Szczur commented on BEAM-315:
---

[~frances] Thanks for explanation. The bug appears with Beam' build in codecs 
and types. The first time I've encountered it I was thinking it may be the 
issue with codec (I was using custom Tuple codec) thus I've spend a lot time 
figuring out it's not really related to key type.

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-122) GlobalWindow and allowedLateness can cause inconsistent timer interpretation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308701#comment-15308701
 ] 

ASF GitHub Bot commented on BEAM-122:
-

GitHub user mshields822 opened a pull request:

https://github.com/apache/incubator-beam/pull/407

[BEAM-122] Forward port changes to backport

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mshields822/incubator-beam beam-122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit 209d49807521a869a96ce73edd44e41dc85ea4dc
Author: Mark Shields 
Date:   2016-05-31T21:44:20Z

Forward port changes to backport




> GlobalWindow and allowedLateness can cause inconsistent timer interpretation 
> -
>
> Key: BEAM-122
> URL: https://issues.apache.org/jira/browse/BEAM-122
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> In ReduceFnRunner we have code such as
>window.getMaxTimestamp().plus(windowingStrategy.getAllowedLateness())
> If window is global then maxTimestamp will be 
> BoundedWindow.TIMESTAMP_MAX_VALUE.
> Meanwhile, timestamps beyond BoundedWindow.TIMESTAMP_MAX_VALUE will be 
> clipped in most runners.
> This could cause the time of an expected timer (eg for garbage collection) to 
> not match the actual time of a fired timer.
> We should either make non-zero allowedLateness on the Global window illegal 
> (probably obnoxious) or explicitly clip it to zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-122] Forward port changes to backport

[mshields822]

GitHub user mshields822 opened a pull request:

https://github.com/apache/incubator-beam/pull/407

[BEAM-122] Forward port changes to backport

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mshields822/incubator-beam beam-122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit 209d49807521a869a96ce73edd44e41dc85ea4dc
Author: Mark Shields 
Date:   2016-05-31T21:44:20Z

Forward port changes to backport




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: [BEAM-155] Use custom Assertions in Spark Strea...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/406

[BEAM-155] Use custom Assertions in Spark Streaming Tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Spark Streaming Side Inputs behave differently than the Beam Model. As
the underlying implementation of PAssert is based on side inputs, this
means that Streaming Spark Tests that use SideInputs as the actuals are
non-portable.

More specifically, this enables pipeline-construction time enforcement
that a Preexisting Side Input must be in a window compatible with the
Global Window (otherwise the side input WindowFn should throw an
exception, e.g. in 
[PartitioningWindowFn](https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PartitioningWindowFn.java#L46)

Modify FlattenStreamingTest, KafkaStreamingTest, and
SimpleStreamingWordCountTest to group all of the contents of the
asserted PCollection into a single key, and assert the contents of that
concatenation, rather than doing so via PAssert and Side Input.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
spark_custom_streaming_assertions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #406


commit ccf3412b06862dddaabbd0869aa0aefca3c77156
Author: Thomas Groh 
Date:   2016-05-31T20:27:43Z

Use custom Assertions in Spark Streaming Tests

Spark Streaming Side Inputs behave differently than the Beam Model. As
the underlying implementation of PAssert is based on side inputs, this
means that Streaming Spark Tests that use SideInputs as the actuals are
non-portable.

Modify FlattenStreamingTest, KafkaStreamingTest, and
SimpleStreamingWordCountTest to group all of the contents of the
asserted PCollection into a single key, and assert the contents of that
concatenation, rather than doing so via PAssert and Side Input.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-155) Support asserting the contents of windows and panes in PAssert

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308622#comment-15308622
 ] 

ASF GitHub Bot commented on BEAM-155:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/406

[BEAM-155] Use custom Assertions in Spark Streaming Tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Spark Streaming Side Inputs behave differently than the Beam Model. As
the underlying implementation of PAssert is based on side inputs, this
means that Streaming Spark Tests that use SideInputs as the actuals are
non-portable.

More specifically, this enables pipeline-construction time enforcement
that a Preexisting Side Input must be in a window compatible with the
Global Window (otherwise the side input WindowFn should throw an
exception, e.g. in 
[PartitioningWindowFn](https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PartitioningWindowFn.java#L46)

Modify FlattenStreamingTest, KafkaStreamingTest, and
SimpleStreamingWordCountTest to group all of the contents of the
asserted PCollection into a single key, and assert the contents of that
concatenation, rather than doing so via PAssert and Side Input.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
spark_custom_streaming_assertions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #406


commit ccf3412b06862dddaabbd0869aa0aefca3c77156
Author: Thomas Groh 
Date:   2016-05-31T20:27:43Z

Use custom Assertions in Spark Streaming Tests

Spark Streaming Side Inputs behave differently than the Beam Model. As
the underlying implementation of PAssert is based on side inputs, this
means that Streaming Spark Tests that use SideInputs as the actuals are
non-portable.

Modify FlattenStreamingTest, KafkaStreamingTest, and
SimpleStreamingWordCountTest to group all of the contents of the
asserted PCollection into a single key, and assert the contents of that
concatenation, rather than doing so via PAssert and Side Input.




> Support asserting the contents of windows and panes in PAssert
> --
>
> Key: BEAM-155
> URL: https://issues.apache.org/jira/browse/BEAM-155
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This consists of reifying the output windows and panes, and running asserts 
> per-window about the contents of panes.
> This includes aggregated matching and final pane matching, e.g.
> PAssert.that(output).byOnTimePane().hasOutputElements(foo, bar);
> // For discarding mode - could have emitted (say) [spam, eggs], [spam], [], 
> [sausage], []
> PAssert.that(output).byFinalPane().hasOutputElements(spam, eggs, sausage, 
> spam);
> // For accumulating mode without late data
> PAssert.that(output).finalPane().containsInAnyOrder(spam, eggs, sausage, 
> spam);
> // For accumulating mode with late data
> PAssert.that(output).finalPane().containsInAnyOrder(foo, 
> bar).mayAlsoContain(baz, rab);
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: Add KEYS file

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/405

Add KEYS file

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
In preparation for the release, add a KEYS file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam KEYS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit 547c9f02e91ca5c5e92a4adae2eb5fd068967f94
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T20:07:48Z

Add KEYS file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: Fix github template and exclude from rat check

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/404


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #404

[davor]

This closes #404


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/00ab9b5b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/00ab9b5b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/00ab9b5b

Branch: refs/heads/master
Commit: 00ab9b5b5ac0f3fe8837b663768ccbef78fdd5d2
Parents: 6c338da 764a923
Author: Davor Bonaci 
Authored: Tue May 31 12:59:30 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 12:59:30 2016 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md | 19 ---
 pom.xml  |  1 +
 2 files changed, 1 insertion(+), 19 deletions(-)
--

[1/2] incubator-beam git commit: Fix github template and exclude from rat check

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6c338da0d -> 00ab9b5b5


Fix github template and exclude from rat check


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/764a9233
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/764a9233
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/764a9233

Branch: refs/heads/master
Commit: 764a9233126804ce88c4fcf9c82f64e06f8ed106
Parents: 6c338da
Author: Jean-Baptiste Onofré 
Authored: Tue May 31 21:53:19 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue May 31 21:53:19 2016 +0200

--
 .github/PULL_REQUEST_TEMPLATE.md | 19 ---
 pom.xml  |  1 +
 2 files changed, 1 insertion(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/764a9233/.github/PULL_REQUEST_TEMPLATE.md
--
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 7db5d32..b17f620 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,22 +1,3 @@
-
-
 Be sure to do all of the following to help us incorporate your contribution
 quickly and easily:
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/764a9233/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 2a47265..ecc379d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -240,6 +240,7 @@
   
   **/target/**/*
   **/dependency-reduced-pom.xml
+  .github/**/*
   **/*.iml
   **/package-list
   **/user.avsc

[GitHub] incubator-beam pull request: Fix github template and exclude from rat check

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/404

Fix github template and exclude from rat check

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam GITHUB-TEMPLATE-FIX

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/404.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #404


commit 764a9233126804ce88c4fcf9c82f64e06f8ed106
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T19:53:19Z

Fix github template and exclude from rat check




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308465#comment-15308465
 ] 

ASF GitHub Bot commented on BEAM-242:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/403

[BEAM-242] Set javadoc less strict around missing return or param sta…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…tements

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit 5c3351c2c72c6402557a9afce36bcacd43fd8d7a
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T19:46:46Z

[BEAM-242] Set javadoc less strict around missing return or param statements




> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-242] Set javadoc less strict around missi...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/403

[BEAM-242] Set javadoc less strict around missing return or param sta…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…tements

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit 5c3351c2c72c6402557a9afce36bcacd43fd8d7a
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T19:46:46Z

[BEAM-242] Set javadoc less strict around missing return or param statements




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-254) Add maven-rat-plugin execution

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-254.
---
Resolution: Fixed

> Add maven-rat-plugin execution
> --
>
> Key: BEAM-254
> URL: https://issues.apache.org/jira/browse/BEAM-254
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> In order to check the license headers and validate we are "Apache compliant", 
> I will add the maven-rat-plugin execution in the main parent POM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-254) Add maven-rat-plugin execution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308449#comment-15308449
 ] 

ASF GitHub Bot commented on BEAM-254:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/297


> Add maven-rat-plugin execution
> --
>
> Key: BEAM-254
> URL: https://issues.apache.org/jira/browse/BEAM-254
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> In order to check the license headers and validate we are "Apache compliant", 
> I will add the maven-rat-plugin execution in the main parent POM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-254] Add apache-rat-plugin execution and fix files with missing license header

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 731c03036 -> 6c338da0d


[BEAM-254] Add apache-rat-plugin execution and fix files with missing license 
header


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e26cf308
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e26cf308
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e26cf308

Branch: refs/heads/master
Commit: e26cf308ec5c83e1238645379526ebef288f1fec
Parents: 731c030
Author: Jean-Baptiste Onofré 
Authored: Fri May 6 09:05:00 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue May 31 19:19:11 2016 +0200

--
 .github/PULL_REQUEST_TEMPLATE.md| 19 +
 .travis.yml | 17 
 README.md   | 19 +
 examples/java/README.md | 19 +
 .../org/apache/beam/examples/complete/README.md | 18 +
 .../org/apache/beam/examples/cookbook/README.md | 18 +
 .../beam/examples/complete/game/README.md   | 18 +
 pom.xml | 42 
 runners/flink/README.md | 19 +
 runners/spark/README.md | 19 +
 .../src/main/resources/beam/checkstyle.xml  | 15 ---
 .../src/main/resources/beam/header-file.txt | 17 
 sdks/java/core/src/main/proto/README.md | 23 ++-
 .../org/apache/beam/sdk/sdk.properties  | 19 -
 sdks/java/extensions/join-library/README.md | 29 +++---
 sdks/java/io/hdfs/README.md | 19 +
 sdks/java/javadoc/README.md | 19 +
 sdks/java/javadoc/overview.html | 26 +---
 .../META-INF/maven/archetype-metadata.xml   | 16 
 .../projects/basic/archetype.properties | 16 
 .../META-INF/maven/archetype-metadata.xml   | 16 
 .../main/resources/archetype-resources/pom.xml  | 17 
 .../projects/basic/archetype.properties | 16 
 .../resources/projects/basic/reference/pom.xml  | 17 
 travis/README.md| 19 +
 travis/test_wordcount.sh| 16 
 26 files changed, 463 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26cf308/.github/PULL_REQUEST_TEMPLATE.md
--
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index b17f620..7db5d32 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,3 +1,22 @@
+
+
 Be sure to do all of the following to help us incorporate your contribution
 quickly and easily:
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26cf308/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 8aad36e..ddf3204 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,3 +1,20 @@
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
 language: java
 
 sudo: false

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26cf308/README.md
--
diff --git a/README.md b/README.md
index 4419464..fa00e82 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,22 @@
+
+
 # Apache Beam
 
 [Apache Beam](http://beam.incubator.apache.org) is a unified model for 
defining both batch and streaming data-parallel processing pipelines, as well 
as a set of language-specific SDKs for constructing pipelines and Runners for 
executing them on distributed processing backends like [Apache 
Spark](http://spark.apache.org/), [Apache Flink](http://flink.apache.org), and 
[Google Cloud Dataflow](http://cloud.google.com/dataflow).

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26cf308/e

[GitHub] incubator-beam pull request: [BEAM-254] Add apache-rat-plugin execution and ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/297


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #297

[davor]

This closes #297


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6c338da0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6c338da0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6c338da0

Branch: refs/heads/master
Commit: 6c338da0da26a10843a0dc868836e66f54bec956
Parents: 731c030 e26cf30
Author: Davor Bonaci 
Authored: Tue May 31 12:41:12 2016 -0700
Committer: Davor Bonaci 
Committed: Tue May 31 12:41:12 2016 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md| 19 +
 .travis.yml | 17 
 README.md   | 19 +
 examples/java/README.md | 19 +
 .../org/apache/beam/examples/complete/README.md | 18 +
 .../org/apache/beam/examples/cookbook/README.md | 18 +
 .../beam/examples/complete/game/README.md   | 18 +
 pom.xml | 42 
 runners/flink/README.md | 19 +
 runners/spark/README.md | 19 +
 .../src/main/resources/beam/checkstyle.xml  | 15 ---
 .../src/main/resources/beam/header-file.txt | 17 
 sdks/java/core/src/main/proto/README.md | 23 ++-
 .../org/apache/beam/sdk/sdk.properties  | 19 -
 sdks/java/extensions/join-library/README.md | 29 +++---
 sdks/java/io/hdfs/README.md | 19 +
 sdks/java/javadoc/README.md | 19 +
 sdks/java/javadoc/overview.html | 26 +---
 .../META-INF/maven/archetype-metadata.xml   | 16 
 .../projects/basic/archetype.properties | 16 
 .../META-INF/maven/archetype-metadata.xml   | 16 
 .../main/resources/archetype-resources/pom.xml  | 17 
 .../projects/basic/archetype.properties | 16 
 .../resources/projects/basic/reference/pom.xml  | 17 
 travis/README.md| 19 +
 travis/test_wordcount.sh| 16 
 26 files changed, 463 insertions(+), 45 deletions(-)
--

[jira] [Created] (BEAM-317) Implement PAssert#containsInAnyOrder and PAssert#satisfies without using Side Inputs

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-317:


 Summary: Implement PAssert#containsInAnyOrder and 
PAssert#satisfies without using Side Inputs
 Key: BEAM-317
 URL: https://issues.apache.org/jira/browse/BEAM-317
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


SideInputs are not supported equally by all runners. Implementing non-empty 
assertions as a main input and a serialized assertion allows runner 
implementors to test their implementations prior to constructing side inputs.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308245#comment-15308245
 ] 

Frances Perry commented on BEAM-315:


The Beam Model (and hence the DataflowRunner) specifies that GoupByKey compares 
for key equality using encoded elements. 

https://github.com/apache/incubator-beam/blob/638cc20719cb314d9e6b29fa79d45edcf4586dce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java#L50

That's also why GroupByKey.validate() ensures the deterministic coder 
requirements -- if the user does a GBK, they must provide a key coder that is 
known to produce the same encoding for equal keys. 

https://github.com/apache/incubator-beam/blob/638cc20719cb314d9e6b29fa79d45edcf4586dce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java#L188
 

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-53) PubSubIO: reimplement in Java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308173#comment-15308173
 ] 

ASF GitHub Bot commented on BEAM-53:


Github user dhalperi closed the pull request at:

https://github.com/apache/incubator-beam/pull/367


> PubSubIO: reimplement in Java
> -
>
> Key: BEAM-53
> URL: https://issues.apache.org/jira/browse/BEAM-53
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Daniel Halperin
>Assignee: Mark Shields
>
> PubSubIO is currently only partially implemented in Java: the 
> DirectPipelineRunner uses a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path 
> implemented in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in 
> a scalable way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than 
> from a message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-53] PubSub: build fixups

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/incubator-beam/pull/367


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: AvroSource: synchronization fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/373


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: AvroSource: synchronization fix

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2ffecfda2 -> 731c03036


AvroSource: synchronization fix


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/20a58873
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/20a58873
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/20a58873

Branch: refs/heads/master
Commit: 20a5887361416db516001aec984a58f7da7b3e50
Parents: 2ffecfd
Author: Dan Halperin 
Authored: Mon May 23 12:57:26 2016 -0700
Committer: Dan Halperin 
Committed: Tue May 31 10:09:43 2016 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/io/AvroSource.java   | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/20a58873/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
index 255199f..9cc0b98 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
@@ -494,7 +494,10 @@ public class AvroSource extends BlockBasedSource {
 // Postcondition: same as above, but for the new current (formerly next) 
block.
 @Override
 public boolean readNextBlock() throws IOException {
-  long startOfNextBlock = currentBlockOffset + currentBlockSizeBytes;
+  long startOfNextBlock;
+  synchronized (progressLock) {
+startOfNextBlock = currentBlockOffset + currentBlockSizeBytes;
+  }
 
   // Before reading the variable-sized block header, record the current 
number of bytes read.
   long preHeaderCount = countStream.getBytesRead();

[2/2] incubator-beam git commit: Closes #373

[dhalperi]

Closes #373


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/731c0303
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/731c0303
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/731c0303

Branch: refs/heads/master
Commit: 731c03036fdc6131624d19b360a6e990d72ff441
Parents: 2ffecfd 20a5887
Author: Dan Halperin 
Authored: Tue May 31 10:09:44 2016 -0700
Committer: Dan Halperin 
Committed: Tue May 31 10:09:44 2016 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/io/AvroSource.java   | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--

[jira] [Updated] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-313:
--
Component/s: runner-spark

> Enable the use of an existing spark context with the SparkPipelineRunner
> 
>
> Key: BEAM-313
> URL: https://issues.apache.org/jira/browse/BEAM-313
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Abbass Marouni
>Assignee: Jean-Baptiste Onofré
>
> The general use case is that the SparkPipelineRunner creates its own Spark 
> context and uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing 
> spark context. This can be interesting for a lot of use cases where the Spark 
> context is managed outside of beam (context reuse, advanced context 
> management, spark job server, ...).
> code sample : 
> https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-316) Support file scheme in TextIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308001#comment-15308001
 ] 

ASF GitHub Bot commented on BEAM-316:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/402

[BEAM-316] Add file scheme support in TextIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Add file sheme support in TextIO.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-316

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 40aa31e5730acdfa5629d2f6f5080bf7e21e
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T16:04:52Z

[BEAM-316] Add file scheme support in TextIO




> Support file scheme in TextIO
> -
>
> Key: BEAM-316
> URL: https://issues.apache.org/jira/browse/BEAM-316
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> When users use {{TextIO}}, most of the time, they provide a full file URI: 
> {{file:/tmp/foo}}. Unfortunately, the {{file}} schema is not supported by 
> {{TextIO}}  and it fails with "No handler found". It's not easy for users to 
> figure it out.
> We should support {{file}} schema to provide better flexibility to users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-316] Add file scheme support in TextIO

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/402

[BEAM-316] Add file scheme support in TextIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Add file sheme support in TextIO.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-316

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 40aa31e5730acdfa5629d2f6f5080bf7e21e
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T16:04:52Z

[BEAM-316] Add file scheme support in TextIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-316) Support file scheme in TextIO

[JIRA]

Jean-Baptiste Onofré created BEAM-316:
-

 Summary: Support file scheme in TextIO
 Key: BEAM-316
 URL: https://issues.apache.org/jira/browse/BEAM-316
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


When users use {{TextIO}}, most of the time, they provide a full file URI: 
{{file:/tmp/foo}}. Unfortunately, the {{file}} schema is not supported by 
{{TextIO}}  and it fails with "No handler found". It's not easy for users to 
figure it out.
We should support {{file}} schema to provide better flexibility to users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307958#comment-15307958
 ] 

Pawel Szczur edited comment on BEAM-315 at 5/31/16 3:50 PM:


I've tried to set {{options.setStreaming(true)}} it seams that items are 
processed even more granularly.



was (Author: pawelszc...@gmail.com):
I've tried to set {options.setStreaming(true)} it seams that items are 
processed even more granularly.


> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307958#comment-15307958
 ] 

Pawel Szczur commented on BEAM-315:
---

I've tried to set {options.setStreaming(true)} it seams that items are 
processed even more granularly.


> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Szczur updated BEAM-315:
--
Comment: was deleted

(was: I've tried {{options.setStreaming(true)}} it caused: 

{code:borderStyle=solid}
Caused by: java.lang.IllegalStateException: No operators defined in 
streaming topology. Cannot execute.
{code})

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307925#comment-15307925
 ] 

Pawel Szczur commented on BEAM-315:
---

I've tried {{options.setStreaming(true)}} it caused: 

{code:borderStyle=solid}
Caused by: java.lang.IllegalStateException: No operators defined in 
streaming topology. Cannot execute.
{code}

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307853#comment-15307853
 ] 

Maximilian Michels commented on BEAM-315:
-

It appears this is only a problem in the batch execution mode. Could you try to 
enable streamingMode on the FlinkPipelineOptions?

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-315:

Summary: GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
FlinkPipelineRunner  (was: GroupByKey/CoGroupByKey doesn't group correctly with 
FlinkPipelineRunner)

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of 
> FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307833#comment-15307833
 ] 

Pawel Szczur edited comment on BEAM-315 at 5/31/16 2:51 PM:


Using bleeding-edge of Beam. I've noticed the problem around last Tuesday, so 
it's there at least that long.

The bug appears with different key types. I've tried: Protocol Buffers message, 
Tuple, Avro and simple types.


was (Author: pawelszc...@gmail.com):
Using bleeding-edge of Beam. I've noticed the problem around last Tuesday, so 
it's there at least that long.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307833#comment-15307833
 ] 

Pawel Szczur edited comment on BEAM-315 at 5/31/16 2:50 PM:


Using bleeding-edge of Beam. I've noticed the problem around last Thursday, so 
it's there at least that long.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.


was (Author: pawelszc...@gmail.com):
Using bleeding-edge of Beam.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307833#comment-15307833
 ] 

Pawel Szczur edited comment on BEAM-315 at 5/31/16 2:51 PM:


Using bleeding-edge of Beam. I've noticed the problem around last Tuesday, so 
it's there at least that long.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.


was (Author: pawelszc...@gmail.com):
Using bleeding-edge of Beam. I've noticed the problem around last Thursday, so 
it's there at least that long.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307833#comment-15307833
 ] 

Pawel Szczur commented on BEAM-315:
---

Using bleeding-edge of Beam.

The bug appears with different key types. I've tried: Protocol Message, Tuple, 
Avro and simple types.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307827#comment-15307827
 ] 

Aljoscha Krettek commented on BEAM-315:
---

I attached a version that uses a {{String}} as key. With this, the results are 
also wrong but "less wrong" than with the {{Key}} class. I think the problem 
with having {{Key}} as a key is that {{AvroCoder.consistentWithEquals()}} is 
{{false}} and the Flink runner uses the serialized bytes to do comparisons. Not 
sure how the Dataflow runner deals with this, though. Also, once data is 
sufficiently large for the bug to appear the pipeline can not be executed on 
the {{DirectPipelineRunner}} or the {{InProcessPipelineRunner}} because both 
fail with a OOM exception.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-315:
--
Attachment: CoGroupPipelineStringKey.java

This is a version of the same program that uses a String key instead of the 
{{Key}} class.

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307796#comment-15307796
 ] 

Maximilian Michels commented on BEAM-315:
-

Thanks for reporting! I'm assuming you're using the latest master?

> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307738#comment-15307738
 ] 

ASF GitHub Bot commented on BEAM-313:
-

GitHub user amarouni opened a pull request:

https://github.com/apache/incubator-beam/pull/401

[BEAM-313] Enable the use of an existing spark context with the 
SparkPipelineRunner

The general use case is that the SparkPipelineRunner creates its own Spark 
context and uses it for the pipeline execution.
Another alternative is to provide the SparkPipelineRunner with an existing 
spark context. This can be interesting for a lot of use cases where the Spark 
context is managed outside of beam (context reuse, advanced context management, 
spark job server, ...).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amarouni/incubator-beam mycbeam313

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit fec04dee06cad9bb6d8914c6ba027a5574c73e41
Author: Abbass MAROUNI 
Date:   2016-05-31T12:45:32Z

[BEAM-313] Enable the use of an existing spark context with the 
SparkPipelineRunner




> Enable the use of an existing spark context with the SparkPipelineRunner
> 
>
> Key: BEAM-313
> URL: https://issues.apache.org/jira/browse/BEAM-313
> Project: Beam
>  Issue Type: New Feature
>Reporter: Abbass Marouni
>Assignee: Jean-Baptiste Onofré
>
> The general use case is that the SparkPipelineRunner creates its own Spark 
> context and uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing 
> spark context. This can be interesting for a lot of use cases where the Spark 
> context is managed outside of beam (context reuse, advanced context 
> management, spark job server, ...).
> code sample : 
> https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-313] Enable the use of an existing spark ...

[amarouni]

GitHub user amarouni opened a pull request:

https://github.com/apache/incubator-beam/pull/401

[BEAM-313] Enable the use of an existing spark context with the 
SparkPipelineRunner

The general use case is that the SparkPipelineRunner creates its own Spark 
context and uses it for the pipeline execution.
Another alternative is to provide the SparkPipelineRunner with an existing 
spark context. This can be interesting for a lot of use cases where the Spark 
context is managed outside of beam (context reuse, advanced context management, 
spark job server, ...).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amarouni/incubator-beam mycbeam313

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit fec04dee06cad9bb6d8914c6ba027a5574c73e41
Author: Abbass MAROUNI 
Date:   2016-05-31T12:45:32Z

[BEAM-313] Enable the use of an existing spark context with the 
SparkPipelineRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Szczur updated BEAM-315:
--
Description: 
Same keys are processed multiple times.

A repo to reproduce the bug:
https://github.com/orian/cogroup-wrong-grouping

Discussion:
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E

Notice: I haven't tested other runners (didn't manage to configure Spark).

  was:
Same keys is processed multiple times.

A repo to reproduce the bug:
https://github.com/orian/cogroup-wrong-grouping

Discussion:
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E

Notice: I haven't tested other runners (didn't manage to configure Spark).


> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Szczur updated BEAM-315:
--
Description: 
Same keys is processed multiple times.

A repo to reproduce the bug:
https://github.com/orian/cogroup-wrong-grouping

Discussion:
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E

Notice: I haven't tested other runners (didn't manage to configure Spark).

  was:
Same keys is processed multiple times.

A repo to reproduce the bug:
https://github.com/orian/cogroup-wrong-grouping

Discussion:
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E


> GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner
> 
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating
>Reporter: Pawel Szczur
>
> Same keys is processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly with FlinkPipelineRunner

[Pawel Szczur (JIRA)]

Pawel Szczur created BEAM-315:
-

 Summary: GroupByKey/CoGroupByKey doesn't group correctly with 
FlinkPipelineRunner
 Key: BEAM-315
 URL: https://issues.apache.org/jira/browse/BEAM-315
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.1.0-incubating
Reporter: Pawel Szczur


Same keys is processed multiple times.

A repo to reproduce the bug:
https://github.com/orian/cogroup-wrong-grouping

Discussion:
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-314) Support zip compression in TextIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307642#comment-15307642
 ] 

ASF GitHub Bot commented on BEAM-314:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/400

[BEAM-314] Add zip compression support in TextIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Add zip compression support in TextIO.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-314

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit ce2dcec24b0c849144b5c2c9d7085983e979bc26
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T12:23:27Z

[BEAM-314] Add zip compression support in TextIO




> Support zip compression in TextIO
> -
>
> Key: BEAM-314
> URL: https://issues.apache.org/jira/browse/BEAM-314
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Now, {{TextIO}} supports Gzip and Bzip2 compression, but not Zip. Even if 
> it's not the same kind of compression (as zip archive may contain several 
> entries), a classic use case is to read "unique" zip compressed file (single 
> entry).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-314] Add zip compression support in TextI...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/400

[BEAM-314] Add zip compression support in TextIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Add zip compression support in TextIO.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-314

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit ce2dcec24b0c849144b5c2c9d7085983e979bc26
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T12:23:27Z

[BEAM-314] Add zip compression support in TextIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-314) Support zip compression in TextIO

[JIRA]

Jean-Baptiste Onofré created BEAM-314:
-

 Summary: Support zip compression in TextIO
 Key: BEAM-314
 URL: https://issues.apache.org/jira/browse/BEAM-314
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


Now, {{TextIO}} supports Gzip and Bzip2 compression, but not Zip. Even if it's 
not the same kind of compression (as zip archive may contain several entries), 
a classic use case is to read "unique" zip compressed file (single entry).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307605#comment-15307605
 ] 

Jean-Baptiste Onofré commented on BEAM-313:
---

Right now, each runner provides its own context, and it's the preferred 
approach as the context is directly handled by the runner.
However, we can imagine to inject the context via the pipeline options.

> Enable the use of an existing spark context with the SparkPipelineRunner
> 
>
> Key: BEAM-313
> URL: https://issues.apache.org/jira/browse/BEAM-313
> Project: Beam
>  Issue Type: New Feature
>Reporter: Abbass Marouni
>Assignee: Jean-Baptiste Onofré
>
> The general use case is that the SparkPipelineRunner creates its own Spark 
> context and uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing 
> spark context. This can be interesting for a lot of use cases where the Spark 
> context is managed outside of beam (context reuse, advanced context 
> management, spark job server, ...).
> code sample : 
> https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-313:
-

Assignee: Jean-Baptiste Onofré

> Enable the use of an existing spark context with the SparkPipelineRunner
> 
>
> Key: BEAM-313
> URL: https://issues.apache.org/jira/browse/BEAM-313
> Project: Beam
>  Issue Type: New Feature
>Reporter: Abbass Marouni
>Assignee: Jean-Baptiste Onofré
>
> The general use case is that the SparkPipelineRunner creates its own Spark 
> context and uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing 
> spark context. This can be interesting for a lot of use cases where the Spark 
> context is managed outside of beam (context reuse, advanced context 
> management, spark job server, ...).
> code sample : 
> https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307533#comment-15307533
 ] 

Ismaël Mejía commented on BEAM-313:
---

[~amitsela] What do you think about this ? The use case is not pure Beam, but 
in practice it makes sense in particular to have something like 
spark-job-server.




> Enable the use of an existing spark context with the SparkPipelineRunner
> 
>
> Key: BEAM-313
> URL: https://issues.apache.org/jira/browse/BEAM-313
> Project: Beam
>  Issue Type: New Feature
>Reporter: Abbass Marouni
>
> The general use case is that the SparkPipelineRunner creates its own Spark 
> context and uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing 
> spark context. This can be interesting for a lot of use cases where the Spark 
> context is managed outside of beam (context reuse, advanced context 
> management, spark job server, ...).
> code sample : 
> https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner

[Abbass Marouni (JIRA)]

Abbass Marouni created BEAM-313:
---

 Summary: Enable the use of an existing spark context with the 
SparkPipelineRunner
 Key: BEAM-313
 URL: https://issues.apache.org/jira/browse/BEAM-313
 Project: Beam
  Issue Type: New Feature
Reporter: Abbass Marouni


The general use case is that the SparkPipelineRunner creates its own Spark 
context and uses it for the pipeline execution.
Another alternative is to provide the SparkPipelineRunner with an existing 
spark context. This can be interesting for a lot of use cases where the Spark 
context is managed outside of beam (context reuse, advanced context management, 
spark job server, ...).

code sample : 
https://github.com/amarouni/incubator-beam/commit/fe0bb517bf0ccde07ef5a61f3e44df695b75f076



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #398

[mxm]

This closes #398


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2ffecfda
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2ffecfda
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2ffecfda

Branch: refs/heads/master
Commit: 2ffecfda2313a609f6db70b942d6d9a8984f464a
Parents: 1cd64bb 9706438
Author: Maximilian Michels 
Authored: Tue May 31 11:13:50 2016 +0200
Committer: Maximilian Michels 
Committed: Tue May 31 11:13:50 2016 +0200

--
 .../translation/FlinkBatchPipelineTranslator.java   | 16 +---
 1 file changed, 1 insertion(+), 15 deletions(-)
--

[GitHub] incubator-beam pull request: [BEAM-270] remove CoGroupByKey transl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/398


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-270) Support Timestamps/Windows in Flink Batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307473#comment-15307473
 ] 

ASF GitHub Bot commented on BEAM-270:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/398


> Support Timestamps/Windows in Flink Batch
> -
>
> Key: BEAM-270
> URL: https://issues.apache.org/jira/browse/BEAM-270
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 0.1.0-incubating
>
>
> Right now, Flink Batch execution does not use {{WindowedValue}} internally, 
> this means that all programs that interact with timestamps/windows will not 
> work. We should just internally wrap everything in {{WindowedValue}} as we do 
> in Flink Streaming. This also makes it very straightforward to add support 
> for windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-270] remove CoGroupByKey translation artifacts

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1cd64bb1a -> 2ffecfda2


[BEAM-270] remove CoGroupByKey translation artifacts

We used to have an optimization for the CoGroupByKey operation with two
inputs. This is no longer the cases after changes to the batch execution
in BEAM-270.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9706438e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9706438e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9706438e

Branch: refs/heads/master
Commit: 9706438e3a1988c05ab16f9c14912af958ef875a
Parents: 36a27f5
Author: Maximilian Michels 
Authored: Mon May 30 14:42:57 2016 +0200
Committer: Maximilian Michels 
Committed: Mon May 30 14:47:36 2016 +0200

--
 .../translation/FlinkBatchPipelineTranslator.java   | 16 +---
 1 file changed, 1 insertion(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9706438e/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
index a19f29d..8f9a37a 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
@@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.runners.TransformTreeNode;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.join.CoGroupByKey;
-import org.apache.beam.sdk.values.PValue;
 
 import org.apache.flink.api.java.DataSet;
 import org.apache.flink.api.java.ExecutionEnvironment;
@@ -134,19 +132,7 @@ public class FlinkBatchPipelineTranslator extends 
FlinkPipelineTranslator {
   return null;
 }
 
-BatchTransformTranslator translator = 
FlinkBatchTransformTranslators.getTranslator(transform);
-
-// No translator known
-if (translator == null) {
-  return null;
-}
-
-// We actually only specialize CoGroupByKey when exactly 2 inputs
-if (transform instanceof CoGroupByKey && node.getInput().expand().size() 
!= 2) {
-  return null;
-}
-
-return translator;
+return FlinkBatchTransformTranslators.getTranslator(transform);
   }
 
   private static String formatNodeName(TransformTreeNode node) {