[GitHub] incubator-beam pull request #1059: Fix Streaming Integration Test Failure in...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1059

Fix Streaming Integration Test Failure in Precommit

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Made some changes in profiles which is used to start integration tests in 
Jenkins pre-commit.

Instead of having two profiles for batch and streaming integration tests 
separately, specify integration tests in execution part of each runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
fix-jenkins-precommit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1059


commit 2bb3f3f19f8d62eef97ee5ebec120c4e3457d6ba
Author: Mark Liu 
Date:   2016-10-06T00:50:05Z

Fix Precommit Integration Test Failure




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-722) Add Display Data to the Python SDK

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-722:
--

 Summary: Add Display Data to the Python SDK
 Key: BEAM-722
 URL: https://issues.apache.org/jira/browse/BEAM-722
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Pablo Estrada
Assignee: Frances Perry


The DisplayData feature has been added to the Java SDK (see blog post 
announcing it: 
https://cloud.google.com/blog/big-data/2016/06/dataflow-updates-see-more-details-about-your-pipelines).
 We need now to add it to the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-626) AvroCoder not deserializing correctly in Kryo

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-626:
-
Assignee: Aviem Zur  (was: Davor Bonaci)

> AvroCoder not deserializing correctly in Kryo
> -
>
> Key: BEAM-626
> URL: https://issues.apache.org/jira/browse/BEAM-626
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Minor
>
> Unlike with Java serialization, when deserializing AvroCoder using Kryo, the 
> resulting AvroCoder is missing all of its transient fields.
> The reason it works with Java serialization is because of the usage of 
> writeReplace and readResolve, which Kryo does not adhere to.
> In ProtoCoder for example there are also unserializable members, the way it 
> is solved there is lazy initializing these members via their getters, so they 
> are initialized in the deserialized object on first call to the member.
> It seems AvroCoder is the only class in Beam to use writeReplace convention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-721) Travis CI fails to run Python tox tests on Mac

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-721:
--

 Summary: Travis CI fails to run Python tox tests on Mac
 Key: BEAM-721
 URL: https://issues.apache.org/jira/browse/BEAM-721
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
 Environment: Mac
Reporter: Pablo Estrada
Assignee: Frances Perry


Some Travis CI runs on Mac are failing because the test script can not find tox.
See: https://travis-ci.org/apache/incubator-beam/jobs/165306424#L86

The travis.yml file does attempt to install tox (See: 
https://github.com/apache/incubator-beam/blob/python-sdk/.travis.yml#L66)

Looking at the logs, it seems that tox is available in a different directory 
(/usr/local), and TOX_HOME is set to $HOME/Library/Python/2.7/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1058: Fixes a bug in avroio_test.py

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/1058

Fixes a bug in avroio_test.py

Fixes a bug in avroio_test.py where we open a binary file without 'b' mode. 
Without this, file can get corrupted in Windows and the test becomes flaky.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
fix_avro_test_windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1058.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1058


commit 6ed258e97a3bde460315b5aef1449c38f80dc564
Author: Chamikara Jayalath 
Date:   2016-10-05T23:23:09Z

Fixes a bug in avroio_test.py where we open a binary file without 'b'
mode. Without this file can get corrupted in Windows and the test becomes 
flaky.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1057: Ignore: test travis

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1057

Ignore: test travis



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam travis-test-archetype

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1057.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1057


commit 06c28bbdec6537d672e2bd27ca71e483bd16a72c
Author: Pei He 
Date:   2016-10-05T22:52:57Z

test travis




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (BEAM-719) Running WindowedWordCount Integration Test in Spark

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-719:
---
Comment: was deleted

(was: As discussed in the mentioned PR, UnboundedReadFromBoundedSource support 
is required, meaning Read.Unbounded support, so this is currently blocked by 
BEAM-658.)

> Running WindowedWordCount Integration Test in Spark
> ---
>
> Key: BEAM-719
> URL: https://issues.apache.org/jira/browse/BEAM-719
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Amit Sela
>
> The purpose of running WindowedWordCountIT in Spark is to have a streaming 
> test pipeline running in Jenkins pre-commit using TestSparkRunner.
> More discussion happened here:
> https://github.com/apache/incubator-beam/pull/1045#issuecomment-251531770



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-719) Running WindowedWordCount Integration Test in Spark

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550096#comment-15550096
 ] 

Amit Sela commented on BEAM-719:


As discussed in the mentioned PR, UnboundedReadFromBoundedSource support is 
required, meaning Read.Unbounded support, so this is currently blocked by 
BEAM-658.

> Running WindowedWordCount Integration Test in Spark
> ---
>
> Key: BEAM-719
> URL: https://issues.apache.org/jira/browse/BEAM-719
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Amit Sela
>
> The purpose of running WindowedWordCountIT in Spark is to have a streaming 
> test pipeline running in Jenkins pre-commit using TestSparkRunner.
> More discussion happened here:
> https://github.com/apache/incubator-beam/pull/1045#issuecomment-251531770



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-719) Running WindowedWordCount Integration Test in Spark

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550095#comment-15550095
 ] 

Amit Sela commented on BEAM-719:


As discussed in the mentioned PR, UnboundedReadFromBoundedSource support is 
required, meaning Read.Unbounded support, so this is currently blocked by 
BEAM-658.

> Running WindowedWordCount Integration Test in Spark
> ---
>
> Key: BEAM-719
> URL: https://issues.apache.org/jira/browse/BEAM-719
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Amit Sela
>
> The purpose of running WindowedWordCountIT in Spark is to have a streaming 
> test pipeline running in Jenkins pre-commit using TestSparkRunner.
> More discussion happened here:
> https://github.com/apache/incubator-beam/pull/1045#issuecomment-251531770



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-540) Dataflow streaming jobs running on windmill do not need data disks

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-540:
--
Assignee: David Rieber  (was: Davor Bonaci)

> Dataflow streaming jobs running on windmill do not need data disks
> --
>
> Key: BEAM-540
> URL: https://issues.apache.org/jira/browse/BEAM-540
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: David Rieber
>Assignee: David Rieber
> Fix For: 0.3.0-incubating
>
>
> Dataflow streaming jobs running on windmill do not need data disks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-540) Dataflow streaming jobs running on windmill do not need data disks

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-540.
---
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Dataflow streaming jobs running on windmill do not need data disks
> --
>
> Key: BEAM-540
> URL: https://issues.apache.org/jira/browse/BEAM-540
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: David Rieber
>Assignee: David Rieber
> Fix For: 0.3.0-incubating
>
>
> Dataflow streaming jobs running on windmill do not need data disks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1056: Static import Assert.assertEquals in Data...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1056


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Static import Assert.assertEquals in DataflowUnboundedReadFromBoundedSourceTest

[lcwik]

Static import Assert.assertEquals in DataflowUnboundedReadFromBoundedSourceTest

This closes #1056


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f27354f7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f27354f7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f27354f7

Branch: refs/heads/master
Commit: f27354f775a2db527cd97e1a4109f063245b44d2
Parents: 5918fed 099fd9c
Author: Luke Cwik 
Authored: Wed Oct 5 14:36:32 2016 -0700
Committer: Luke Cwik 
Committed: Wed Oct 5 14:36:32 2016 -0700

--
 .../internal/DataflowUnboundedReadFromBoundedSourceTest.java  | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--

[1/2] incubator-beam git commit: Static import Assert.assertEquals

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5918fed8a -> f27354f77


Static import Assert.assertEquals


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/099fd9c7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/099fd9c7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/099fd9c7

Branch: refs/heads/master
Commit: 099fd9c740bac1bed5d3b7227c2f5b895808e99a
Parents: 5918fed
Author: Scott Wegner 
Authored: Wed Oct 5 13:58:40 2016 -0700
Committer: Scott Wegner 
Committed: Wed Oct 5 13:58:40 2016 -0700

--
 .../internal/DataflowUnboundedReadFromBoundedSourceTest.java  | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/099fd9c7/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/internal/DataflowUnboundedReadFromBoundedSourceTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/internal/DataflowUnboundedReadFromBoundedSourceTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/internal/DataflowUnboundedReadFromBoundedSourceTest.java
index d9e3558..3a88935 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/internal/DataflowUnboundedReadFromBoundedSourceTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/internal/DataflowUnboundedReadFromBoundedSourceTest.java
@@ -18,12 +18,13 @@
 
 package org.apache.beam.runners.dataflow.internal;
 
+import static org.junit.Assert.assertEquals;
+
 import java.io.IOException;
 import java.util.List;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.junit.Assert;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
@@ -38,7 +39,7 @@ public class DataflowUnboundedReadFromBoundedSourceTest {
 DataflowUnboundedReadFromBoundedSource read = new
 DataflowUnboundedReadFromBoundedSource<>(new NoopNamedSource());
 
-Assert.assertEquals("Read(NoopNamedSource)", read.getKindString());
+assertEquals("Read(NoopNamedSource)", read.getKindString());
   }
 
   @Test
@@ -47,7 +48,7 @@ public class DataflowUnboundedReadFromBoundedSourceTest {
 DataflowUnboundedReadFromBoundedSource read = new
 DataflowUnboundedReadFromBoundedSource<>(anonSource);
 
-Assert.assertEquals("Read(AnonymousSource)", read.getKindString());
+assertEquals("Read(AnonymousSource)", read.getKindString());
   }
 
   /** Source implementation only useful for its identity. */

[jira] [Created] (BEAM-720) Running WindowedWordCount Integration Test in Flink

[Mark Liu (JIRA)]

Mark Liu created BEAM-720:
-

 Summary: Running WindowedWordCount Integration Test in Flink
 Key: BEAM-720
 URL: https://issues.apache.org/jira/browse/BEAM-720
 Project: Beam
  Issue Type: Improvement
Reporter: Mark Liu
Assignee: Aljoscha Krettek


In order to have coverage of streaming pipeline test in pre-commit, it's 
important to have TestFlinkRunner to be able to run WindowedWordCountIT 
successfully. 

Relevant works in TestDataflowRunner:
https://github.com/apache/incubator-beam/pull/1045



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-719) Running WindowedWordCount Integration Test in Spark

[Mark Liu (JIRA)]

Mark Liu created BEAM-719:
-

 Summary: Running WindowedWordCount Integration Test in Spark
 Key: BEAM-719
 URL: https://issues.apache.org/jira/browse/BEAM-719
 Project: Beam
  Issue Type: Improvement
Reporter: Mark Liu
Assignee: Amit Sela


The purpose of running WindowedWordCountIT in Spark is to have a streaming test 
pipeline running in Jenkins pre-commit using TestSparkRunner.

More discussion happened here:
https://github.com/apache/incubator-beam/pull/1045#issuecomment-251531770



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1056: Static import Assert.assertEquals in Data...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1056

Static import Assert.assertEquals in 
DataflowUnboundedReadFromBoundedSourceTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam assert-staticimport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1056.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1056


commit 099fd9c740bac1bed5d3b7227c2f5b895808e99a
Author: Scott Wegner 
Date:   2016-10-05T20:58:40Z

Static import Assert.assertEquals




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #863

[kenn]

This closes #863


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5918fed8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5918fed8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5918fed8

Branch: refs/heads/master
Commit: 5918fed8a6e8c7999a5981c47effac1591758d1e
Parents: e46127e 8d659fe
Author: Kenneth Knowles 
Authored: Wed Oct 5 13:05:20 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Oct 5 13:05:20 2016 -0700

--
 .../beam/runners/direct/ExecutorServiceParallelExecutor.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Place fired timers in the Global Window

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e46127e5a -> 5918fed8a


Place fired timers in the Global Window

The runner is theoretically permitted to drop timers it recieves if they
are not present in any window. This ensures that any element provided to
any TransformEvaluator by the runner (not user-assigned windows) is in
at least one window.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d659fec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d659fec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d659fec

Branch: refs/heads/master
Commit: 8d659fec13982c982d58d8abd1007197bc443c61
Parents: 087dcef
Author: Thomas Groh 
Authored: Mon Aug 22 13:27:33 2016 -0700
Committer: Thomas Groh 
Committed: Wed Oct 5 10:00:50 2016 -0700

--
 .../beam/runners/direct/ExecutorServiceParallelExecutor.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8d659fec/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 9e11f6d..53e03c8 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -423,7 +423,7 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
   CommittedBundle bundle =
   evaluationContext
   .createKeyedBundle(keyTimers.getKey(), (PCollection) 
transform.getInput())
-  .add(WindowedValue.valueInEmptyWindows(work))
+  .add(WindowedValue.valueInGlobalWindow(work))
   .commit(evaluationContext.now());
   scheduleConsumption(transform, bundle, new 
TimerIterableCompletionCallback(delivery));
   state.set(ExecutorState.ACTIVE);

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1273

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-528) Add @experimental annotations

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay closed BEAM-528.

   Resolution: Fixed
Fix Version/s: Not applicable

> Add @experimental annotations 
> --
>
> Key: BEAM-528
> URL: https://issues.apache.org/jira/browse/BEAM-528
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: María GH
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>
> Experimental/deprecation warnings: use the warnings standard module in 
> conjunction with decorators as described here:
> https://docs.python.org/2/library/warnings.html
> Some code sample for a deprecated decorator that is kinda/sorta similar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1272

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-716) Migrate JmsIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-716:
--

 Summary: Migrate JmsIO to use AutoValue to reduce boilerplate
 Key: BEAM-716
 URL: https://issues.apache.org/jira/browse/BEAM-716
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Luke Cwik
Assignee: James Malone
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-718) Migrate KinesisIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-718:
--

 Summary: Migrate KinesisIO to use AutoValue to reduce boilerplate
 Key: BEAM-718
 URL: https://issues.apache.org/jira/browse/BEAM-718
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Luke Cwik
Assignee: James Malone
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-717) Migrate KafkaIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-717:
--

 Summary: Migrate KafkaIO to use AutoValue to reduce boilerplate
 Key: BEAM-717
 URL: https://issues.apache.org/jira/browse/BEAM-717
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Luke Cwik
Assignee: James Malone
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549629#comment-15549629
 ] 

Eugene Kirpichov commented on BEAM-714:
---

This is being done in https://github.com/apache/incubator-beam/pull/1031

> Migrate DatastoreV1 to use AutoValue to reduce boilerplate
> --
>
> Key: BEAM-714
> URL: https://issues.apache.org/jira/browse/BEAM-714
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-715) Migrate AvroHDFSFileSource/HDFSFileSource/HDFSFileSink to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-715:
--

 Summary: Migrate AvroHDFSFileSource/HDFSFileSource/HDFSFileSink to 
use AutoValue to reduce boilerplate
 Key: BEAM-715
 URL: https://issues.apache.org/jira/browse/BEAM-715
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Luke Cwik
Assignee: James Malone
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-714:
--

 Summary: Migrate DatastoreV1 to use AutoValue to reduce boilerplate
 Key: BEAM-714
 URL: https://issues.apache.org/jira/browse/BEAM-714
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-712) Migrate BigQueryIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-712:
--

 Summary: Migrate BigQueryIO to use AutoValue to reduce boilerplate
 Key: BEAM-712
 URL: https://issues.apache.org/jira/browse/BEAM-712
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-713) Migrate BigTableIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-713:
--

 Summary: Migrate BigTableIO to use AutoValue to reduce boilerplate
 Key: BEAM-713
 URL: https://issues.apache.org/jira/browse/BEAM-713
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-711) Migrate XmlSource/XmlSink to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-711:
--

 Summary: Migrate XmlSource/XmlSink to use AutoValue to reduce 
boilerplate
 Key: BEAM-711
 URL: https://issues.apache.org/jira/browse/BEAM-711
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-710) Migrate Read/Write to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-710:
---
Summary: Migrate Read/Write to use AutoValue to reduce boilerplate  (was: 
Migrate Read to use AutoValue to reduce boilerplate)

> Migrate Read/Write to use AutoValue to reduce boilerplate
> -
>
> Key: BEAM-710
> URL: https://issues.apache.org/jira/browse/BEAM-710
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-709) Migrate CountingSource/CountingInput to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-709:
---
Summary: Migrate CountingSource/CountingInput to use AutoValue to reduce 
boilerplate  (was: Migrate CountingSource to use AutoValue to reduce 
boilerplate)

> Migrate CountingSource/CountingInput to use AutoValue to reduce boilerplate
> ---
>
> Key: BEAM-709
> URL: https://issues.apache.org/jira/browse/BEAM-709
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-707) Migrate PubsubIO/PubsubUnboundedSource/PubsubUnboundedSink to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-707:
---
Summary: Migrate PubsubIO/PubsubUnboundedSource/PubsubUnboundedSink to use 
AutoValue to reduce boilerplate  (was: Migrate PubsubIO to use AutoValue to 
reduce boilerplate)

> Migrate PubsubIO/PubsubUnboundedSource/PubsubUnboundedSink to use AutoValue 
> to reduce boilerplate
> -
>
> Key: BEAM-707
> URL: https://issues.apache.org/jira/browse/BEAM-707
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-710) Migrate Read to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-710:
--

 Summary: Migrate Read to use AutoValue to reduce boilerplate
 Key: BEAM-710
 URL: https://issues.apache.org/jira/browse/BEAM-710
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-709) Migrate CountingSource to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-709:
--

 Summary: Migrate CountingSource to use AutoValue to reduce 
boilerplate
 Key: BEAM-709
 URL: https://issues.apache.org/jira/browse/BEAM-709
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-708) Migrate BoundedReadFromUnboundedSource to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-708:
--

 Summary: Migrate BoundedReadFromUnboundedSource to use AutoValue 
to reduce boilerplate
 Key: BEAM-708
 URL: https://issues.apache.org/jira/browse/BEAM-708
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-707) Migrate PubsubIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-707:
--

 Summary: Migrate PubsubIO to use AutoValue to reduce boilerplate
 Key: BEAM-707
 URL: https://issues.apache.org/jira/browse/BEAM-707
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-706) Migrate TextIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-706:
--

 Summary: Migrate TextIO to use AutoValue to reduce boilerplate
 Key: BEAM-706
 URL: https://issues.apache.org/jira/browse/BEAM-706
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.
See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-705) Migrate AvroIO to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-705:
--

 Summary: Migrate AvroIO to use AutoValue to reduce boilerplate
 Key: BEAM-705
 URL: https://issues.apache.org/jira/browse/BEAM-705
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci
Priority: Minor


Use the AutoValue functionality to reduce boilerplate.

See this PR for an example:
https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-561) Add WindowedWordCountIT

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549563#comment-15549563
 ] 

ASF GitHub Bot commented on BEAM-561:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1045


> Add WindowedWordCountIT
> ---
>
> Key: BEAM-561
> URL: https://issues.apache.org/jira/browse/BEAM-561
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Mark Liu
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-561] Add Streaming IT in Jenkins Pre-commit

[dhalperi]

[BEAM-561] Add Streaming IT in Jenkins Pre-commit


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f028f1c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f028f1c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f028f1c0

Branch: refs/heads/master
Commit: f028f1c07c9a4229020c082b8a367af68a1ff531
Parents: 565319b
Author: Mark Liu 
Authored: Tue Oct 4 12:43:46 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 5 11:21:59 2016 -0700

--
 examples/java/pom.xml | 63 ++
 1 file changed, 63 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f028f1c0/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 6b1b7ce..0dd8d6d 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -244,6 +244,69 @@
 
   
 
+
+
+  jenkins-precommit-streaming
+  
+WindowedWordCountIT
+false
+true
+  
+  
+
+  
+org.apache.maven.plugins
+maven-failsafe-plugin
+
+  false
+  true
+
+
+  
+direct-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempLocation=gs://temp-storage-for-end-to-end-tests",
+  "--runner=org.apache.beam.runners.direct.DirectRunner"
+  ]
+
+  
+
+  
+  
+dataflow-runner-integration-tests
+
+  integration-test
+  verify
+
+
+  all
+  4
+  
+
+  [
+  "--project=apache-beam-testing",
+  "--tempRoot=gs://temp-storage-for-end-to-end-tests",
+  
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner"
+  ]
+
+  
+
+  
+
+  
+
+  
+
   
 
   

[GitHub] incubator-beam pull request #1045: [BEAM-561] Add Streaming IT to Jenkins Pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1045


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #1045

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 565319bd7 -> e46127e5a


Closes #1045


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e46127e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e46127e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e46127e5

Branch: refs/heads/master
Commit: e46127e5a210173e89f2168c3a9aad8c09443c73
Parents: 565319b f028f1c
Author: Dan Halperin 
Authored: Wed Oct 5 11:21:59 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 5 11:21:59 2016 -0700

--
 examples/java/pom.xml | 63 ++
 1 file changed, 63 insertions(+)
--

[jira] [Commented] (BEAM-259) Execute selected RunnableOnService tests with Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549524#comment-15549524
 ] 

ASF GitHub Bot commented on BEAM-259:
-

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1055

[BEAM-259] Enable RunnableOnService for batch.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Handle empty Flatten for bounded.

Spark will bubble out a SparkException for user code failure, so this won't 
catch. Asserting on the
error message should be good enough.

outputWithTimestamp should handle start/finishBundle as well.

Explode WindowedValues before processing.

sideOutputWithTimestamp to address start/finishBundle.

SideInput with windows.

Unused for now, remove.

Take sideInput window startegy into account, for combine as well.

reduce code duplication.

Spark combine support.

reuse code where possible.

Expose sideInputs and insertDefault in Combine.Globally for direct 
translation.

Direct translation of Combine.Globally into Spark's aggregate function.

Make default run with 4 cores by default - makes tests run with multiple 
threads, but not too many.

SideInputReader for the Spark runner.

A common abstraction for Keyed and Global implementation.

Implement Combine.Globally via Spark's aggregate.

runnable-on-service profile doesn't need pluginManagement.

Removing test as it does not follow a deterministic combine implementation.

Context reuse is mostly for testing. To avoid a test failure that will stop 
the context and fail all
following tests we need to recreate the context if it's stopped as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1055.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1055


commit 53ed9951203da7f7e2cff0ef4d9ec64f37a0f29d
Author: Sela 
Date:   2016-09-23T10:32:28Z

Enable RunnableOnService for batch.

Handle empty Flatten for bounded.

Spark will bubble out a SparkException for user code failure, so this won't 
catch. Asserting on the
error message should be good enough.

outputWithTimestamp should handle start/finishBundle as well.

Explode WindowedValues before processing.

sideOutputWithTimestamp to address start/finishBundle.

SideInput with windows.

Unused for now, remove.

Take sideInput window startegy into account, for combine as well.

reduce code duplication.

Spark combine support.

reuse code where possible.

Expose sideInputs and insertDefault in Combine.Globally for direct 
translation.

Direct translation of Combine.Globally into Spark's aggregate function.

Make default run with 4 cores by default - makes tests run with multiple 
threads, but not too many.

SideInputReader for the Spark runner.

A common abstraction for Keyed and Global implementation.

Implement Combine.Globally via Spark's aggregate.

runnable-on-service profile doesn't need pluginManagement.

Removing test as it does not follow a deterministic combine implementation.

Context reuse is mostly for testing. To avoid a test failure that will stop 
the context and fail all
following tests we need to recreate the context if it's stopped as well.




> Execute selected RunnableOnService tests with Spark runner
> --
>
> Key: BEAM-259
> URL: https://issues.apache.org/jira/browse/BEAM-259
> Project: Beam
>  Issue Type: Test
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Amit Sela
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1055: [BEAM-259] Enable RunnableOnService for b...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1055

[BEAM-259] Enable RunnableOnService for batch.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Handle empty Flatten for bounded.

Spark will bubble out a SparkException for user code failure, so this won't 
catch. Asserting on the
error message should be good enough.

outputWithTimestamp should handle start/finishBundle as well.

Explode WindowedValues before processing.

sideOutputWithTimestamp to address start/finishBundle.

SideInput with windows.

Unused for now, remove.

Take sideInput window startegy into account, for combine as well.

reduce code duplication.

Spark combine support.

reuse code where possible.

Expose sideInputs and insertDefault in Combine.Globally for direct 
translation.

Direct translation of Combine.Globally into Spark's aggregate function.

Make default run with 4 cores by default - makes tests run with multiple 
threads, but not too many.

SideInputReader for the Spark runner.

A common abstraction for Keyed and Global implementation.

Implement Combine.Globally via Spark's aggregate.

runnable-on-service profile doesn't need pluginManagement.

Removing test as it does not follow a deterministic combine implementation.

Context reuse is mostly for testing. To avoid a test failure that will stop 
the context and fail all
following tests we need to recreate the context if it's stopped as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1055.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1055


commit 53ed9951203da7f7e2cff0ef4d9ec64f37a0f29d
Author: Sela 
Date:   2016-09-23T10:32:28Z

Enable RunnableOnService for batch.

Handle empty Flatten for bounded.

Spark will bubble out a SparkException for user code failure, so this won't 
catch. Asserting on the
error message should be good enough.

outputWithTimestamp should handle start/finishBundle as well.

Explode WindowedValues before processing.

sideOutputWithTimestamp to address start/finishBundle.

SideInput with windows.

Unused for now, remove.

Take sideInput window startegy into account, for combine as well.

reduce code duplication.

Spark combine support.

reuse code where possible.

Expose sideInputs and insertDefault in Combine.Globally for direct 
translation.

Direct translation of Combine.Globally into Spark's aggregate function.

Make default run with 4 cores by default - makes tests run with multiple 
threads, but not too many.

SideInputReader for the Spark runner.

A common abstraction for Keyed and Global implementation.

Implement Combine.Globally via Spark's aggregate.

runnable-on-service profile doesn't need pluginManagement.

Removing test as it does not follow a deterministic combine implementation.

Context reuse is mostly for testing. To avoid a test failure that will stop 
the context and fail all
following tests we need to recreate the context if it's stopped as well.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1271

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-259) Execute selected RunnableOnService tests with Spark runner

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549371#comment-15549371
 ] 

Amit Sela commented on BEAM-259:


While this ticket is talking about "enabling" the ROS tests for the Spark 
runner. This turned out to be a work to make those tests pass (batch).
Generally, the runner didn't handle windows via the Beam SDK, and so it had 
many issues with windows, and sideInputs.
On top of that, the time the runner spent "unplugged" of those tests, some 
additional advancements in the SDK were missed as well.

The following PR will introduce changes and additions that provide a full 
support for the Beam model in a batch execution.

> Execute selected RunnableOnService tests with Spark runner
> --
>
> Key: BEAM-259
> URL: https://issues.apache.org/jira/browse/BEAM-259
> Project: Beam
>  Issue Type: Test
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Amit Sela
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-564) Update source framework so that remaining and consumed number of split points can be reported

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549324#comment-15549324
 ] 

ASF GitHub Bot commented on BEAM-564:
-

Github user chamikaramj closed the pull request at:

https://github.com/apache/incubator-beam/pull/881


> Update source framework so that remaining and consumed number of split points 
> can be reported
> -
>
> Key: BEAM-564
> URL: https://issues.apache.org/jira/browse/BEAM-564
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> We have to update Python SDK source framework so that sources can report 
> consumed and remaining number of split points. Runners can use this 
> information to determine how many times a given source can be split into and 
> parallelize reading accordingly.
> Corresponding API for JAVA SDK is here:
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java#L258



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #881: [BEAM-564] Updates sources to report consu...

[chamikaramj]

Github user chamikaramj closed the pull request at:

https://github.com/apache/incubator-beam/pull/881


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-547) Align Python SDK version with Maven

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549295#comment-15549295
 ] 

ASF GitHub Bot commented on BEAM-547:
-

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/855


> Align Python SDK version with Maven
> ---
>
> Key: BEAM-547
> URL: https://issues.apache.org/jira/browse/BEAM-547
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: Sergio Fernández
>Assignee: Frances Perry
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In BEAM-378 we've integrated the Python SDK in the main Maven build. 
> Initially I wanted to also align versions, but after discussing it with 
> [~silv...@google.com] we kept that aside for the moment. 
> Closing [PR #537|https://github.com/apache/incubator-beam/pull/537] [~altay] 
> brings the issue back. So it may make sense to revisit that idea.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #855: [BEAM-547] Read version information from t...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/855


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-175) Don't leak garbage collection timers in GlobalWindow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549289#comment-15549289
 ] 

ASF GitHub Bot commented on BEAM-175:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/139


> Don't leak garbage collection timers in GlobalWindow
> 
>
> Key: BEAM-175
> URL: https://issues.apache.org/jira/browse/BEAM-175
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> Consider the  transform:
>   Window
> .into(new GlobalWindows())
> .triggering(
>   Repeatedly.forever(
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...)))
> .discardingFiredPanes()
> This is a common idiom for 'process elements bunched by arrival time'.
> Currently we create an end-of-window timer per key, which clearly will only 
> fire if the pipeline is drained.
> Better would be to avoid creating end-of-window timers if there's no state 
> which needs to be processed at end-of-window (ie at drain if the Global 
> window).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-523) Minor typo in aggregator_test.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549290#comment-15549290
 ] 

ASF GitHub Bot commented on BEAM-523:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/798


> Minor typo in aggregator_test.py
> 
>
> Key: BEAM-523
> URL: https://issues.apache.org/jira/browse/BEAM-523
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Priority: Trivial
>  Labels: starter
> Fix For: 0.3.0-incubating
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> aggregators is repeatedly misspelled as aggeregators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-160) Port 'NexMark Queries' to Beam for use as integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549288#comment-15549288
 ] 

ASF GitHub Bot commented on BEAM-160:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/366


> Port 'NexMark Queries' to Beam for use as integration test
> --
>
> Key: BEAM-160
> URL: https://issues.apache.org/jira/browse/BEAM-160
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> A while back we implemented the 'queries' from
>   http://datalab.cs.pdx.edu/niagara/NEXMark/
> as Gooogle Dataflow pipelines. We found them useful
> for uncovering performance problems with the sdk, our runners,
> and our service. Many of those problems only manifested under
> high load, multi-day runs, or with high 'backlog' on the incoming
> pub/sub subscriptions.
> We thus think they would be useful for other runners.
> Disclaimer: Though the original 'queries' were proposed as a way to
> benchmark 'continuous SQL' implementations, we have so far only
> used them for internal A/B and regression testing and have not validated
> them as representative of customer workloads. We would thus discourage their 
> use for competitive benchmarks without more work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-548) Combine$PerKeyWithHotKeyFanout should propagate Display Data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549287#comment-15549287
 ] 

ASF GitHub Bot commented on BEAM-548:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/847


> Combine$PerKeyWithHotKeyFanout should propagate Display Data
> 
>
> Key: BEAM-548
> URL: https://issues.apache.org/jira/browse/BEAM-548
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Ben Chambers
>Assignee: Scott Wegner
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> When using hot key combines, we produce two anonymous CombineFn's that wrap 
> the original CombineFn. These currently suppress the display data of the 
> delegate, instead showing up as "Combine$PerKeyWithHotKeyFanout$1".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #656: Removed BatchWrite from DataflowRunner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #139: [BEAM-175][BEAM-32] Don't leak state in Gl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/139


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #798: [BEAM-523] Fixed typo in aggregator_test.p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/798


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #366: [BEAM-160] NexMark

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/366


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #847: [BEAM-548] Fixed Combine display data

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/847


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-beam git commit: Empty commit to close some stale PRs

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 087dcef1e -> 565319bd7


Empty commit to close some stale PRs

Close #139 - abandoned
Close #366 - abandoned
Close #656 - abandoned
Close #798 - merged to Python
Close #847 - merged as part of #906


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/565319bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/565319bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/565319bd

Branch: refs/heads/master
Commit: 565319bd790d8b2b60fbdb3e2f0a72c82b4cbbb3
Parents: 087dcef
Author: Dan Halperin 
Authored: Wed Oct 5 09:43:02 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 5 09:43:02 2016 -0700

--

--

[jira] [Created] (BEAM-704) KafkaIO should handle "latest offset" evenly, and persist it as part of the CheckpointMark.

[Amit Sela (JIRA)]

Amit Sela created BEAM-704:
--

 Summary: KafkaIO should handle "latest offset" evenly, and persist 
it as part of the CheckpointMark.
 Key: BEAM-704
 URL: https://issues.apache.org/jira/browse/BEAM-704
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Amit Sela
Assignee: James Malone


Currently, the KafkaIO (when configured to "latest") will check the latest 
offset on the worker. This means that each worker sees a "different" latest for 
the time it checks for the partitions assigned to it.
This also means that if a worker fails before starting to read, and new 
messages were added in between, they would be missed.

I think we should consider checking the offsets (could be the same for 
"earliest") when running initialSplits (that's how Spark does that as well, one 
call from the driver for all topic-partitions).
I'd also suggest we persist the latest offset as part of the CheckpointMark so 
that once latest is set, it is remembered until new messages arrive and it 
doesn't need to be resolved again (and if there were new messages available 
they won't be missed upon failure).

For Spark this is even more important as state is passed in-between 
micro-batches and sparse partitions may skip messages until a message finally 
arrives within the read time-frame. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1270

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-698) Use AutoValue and deal with Document instead of String in MongoDbIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548115#comment-15548115
 ] 

ASF GitHub Bot commented on BEAM-698:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1054

[BEAM-698] Use AutoValue in MongoDB GridFS

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam 
BEAM-698-AUTOVALUE_GRIDFS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1054.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1054


commit 83d1ed7c06adf35a791c1fee96cd86f73424
Author: Jean-Baptiste Onofré 
Date:   2016-10-03T06:02:07Z

[BEAM-698] Use AutoValue in MongoDB GridFS




> Use AutoValue and deal with Document instead of String in MongoDbIO
> ---
>
> Key: BEAM-698
> URL: https://issues.apache.org/jira/browse/BEAM-698
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1054: [BEAM-698] Use AutoValue in MongoDB GridF...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1054

[BEAM-698] Use AutoValue in MongoDB GridFS

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam 
BEAM-698-AUTOVALUE_GRIDFS

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1054.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1054


commit 83d1ed7c06adf35a791c1fee96cd86f73424
Author: Jean-Baptiste Onofré 
Date:   2016-10-03T06:02:07Z

[BEAM-698] Use AutoValue in MongoDB GridFS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-570) Update AvroSource to support more compression types

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-570:

Assignee: (was: Frances Perry)

> Update AvroSource to support more compression types
> ---
>
> Key: BEAM-570
> URL: https://issues.apache.org/jira/browse/BEAM-570
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>
> Python AvroSource [1] currently only support 'deflate' compression. We should 
> update it to support other compression types supported by the Avro library 
> (e.g.: snappy, bzip2).
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/avroio.py



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-570) Update AvroSource to support more compression types

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-570:

Assignee: Frances Perry  (was: Chamikara Jayalath)

> Update AvroSource to support more compression types
> ---
>
> Key: BEAM-570
> URL: https://issues.apache.org/jira/browse/BEAM-570
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Frances Perry
>
> Python AvroSource [1] currently only support 'deflate' compression. We should 
> update it to support other compression types supported by the Avro library 
> (e.g.: snappy, bzip2).
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/avroio.py



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)