Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1733

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1733

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #1996

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1732

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1732

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #1995

2016-12-05 Thread Apache Jenkins Server 
See

[3/3] incubator-beam git commit: This closes #1493

2016-12-05 Thread kenn 
This closes #1493


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1efda59a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1efda59a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1efda59a

Branch: refs/heads/master
Commit: 1efda59abcaf43a96b5835eb23c2f7951f71663c
Parents: b8e93ff 3434e8a
Author: Kenneth Knowles 
Authored: Mon Dec 5 20:54:19 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 5 20:54:19 2016 -0800

--
 .../apache/beam/sdk/transforms/DoFnTester.java  |  22 ++-
 .../sdk/util/state/InMemoryTimerInternals.java  | 163 ++-
 .../beam/sdk/util/state/TimerCallback.java  |   3 +
 .../util/state/InMemoryTimerInternalsTest.java  |  93 ---
 4 files changed, 204 insertions(+), 77 deletions(-)
--

[1/3] incubator-beam git commit: Deprecate TimerCallback and InMemoryTimerInternals methods using it. Instead separate advancing watermarks and removing eligible timers.

2016-12-05 Thread kenn 
Repository: incubator-beam
Updated Branches:
  refs/heads/master b8e93fff8 -> 1efda59ab


Deprecate TimerCallback and InMemoryTimerInternals methods using it.
Instead separate advancing watermarks and removing eligible timers.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e9d835ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e9d835ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e9d835ee

Branch: refs/heads/master
Commit: e9d835eeb6007ccc86193efbb78570cfc52e665d
Parents: b8e93ff
Author: Sam Whittle 
Authored: Thu Dec 1 22:51:37 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 5 20:54:18 2016 -0800

--
 .../apache/beam/sdk/transforms/DoFnTester.java  |  22 ++-
 .../sdk/util/state/InMemoryTimerInternals.java  | 155 ++-
 .../beam/sdk/util/state/TimerCallback.java  |   3 +
 .../util/state/InMemoryTimerInternalsTest.java  |  75 ++---
 4 files changed, 183 insertions(+), 72 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e9d835ee/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 9f32aec..43896c5 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -52,7 +52,6 @@ import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.InMemoryStateInternals;
 import org.apache.beam.sdk.util.state.InMemoryTimerInternals;
 import org.apache.beam.sdk.util.state.StateInternals;
-import org.apache.beam.sdk.util.state.TimerCallback;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.apache.beam.sdk.values.TupleTag;
@@ -536,19 +535,14 @@ public class DoFnTester implements 
AutoCloseable {
 return extractAggregatorValue(agg.getName(), agg.getCombineFn());
   }
 
-  private static TimerCallback collectInto(final 
List firedTimers) {
-return new TimerCallback() {
-  @Override
-  public void onTimer(TimerInternals.TimerData timer) throws Exception {
-firedTimers.add(timer);
-  }
-};
-  }
-
   public List advanceInputWatermark(Instant 
newWatermark) {
 try {
+  timerInternals.advanceInputWatermark(newWatermark);
   final List firedTimers = new ArrayList<>();
-  timerInternals.advanceInputWatermark(collectInto(firedTimers), 
newWatermark);
+  TimerInternals.TimerData timer;
+  while ((timer = timerInternals.removeNextEventTimer()) != null) {
+firedTimers.add(timer);
+  }
   return firedTimers;
 } catch (Exception e) {
   throw new RuntimeException(e);
@@ -557,8 +551,12 @@ public class DoFnTester implements 
AutoCloseable {
 
   public List advanceProcessingTime(Instant 
newProcessingTime) {
 try {
+  timerInternals.advanceProcessingTime(newProcessingTime);
   final List firedTimers = new ArrayList<>();
-  timerInternals.advanceProcessingTime(collectInto(firedTimers), 
newProcessingTime);
+  TimerInternals.TimerData timer;
+  while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
+firedTimers.add(timer);
+  }
   return firedTimers;
 } catch (Exception e) {
   throw new RuntimeException(e);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e9d835ee/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
index 60a90f5..60c4a96 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
@@ -46,6 +46,9 @@ public class InMemoryTimerInternals implements TimerInternals 
{
   /** Pending processing time timers, in timestamp order. */
   private PriorityQueue processingTimers = new PriorityQueue<>(11);
 
+  /** Pending synchronized processing time timers, in timestamp order. */
+  private PriorityQueue synchronizedProcessingTimers = new 
PriorityQueue<>(11);
+
   /** Current input watermark. */
   @Nullable private Instant inputWatermarkTime = 
BoundedWindow.TIMESTAMP_MIN_VALUE;
 
@@ -76,9

[2/3] incubator-beam git commit: Add test for InMemoryTimerInternals synchronized processing time timers. Ensure that processing time and synchronized processing timer are not null.

2016-12-05 Thread kenn 
Add test for InMemoryTimerInternals synchronized processing time timers.
Ensure that processing time and synchronized processing timer are not null.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3434e8a4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3434e8a4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3434e8a4

Branch: refs/heads/master
Commit: 3434e8a434dea2f0ba5bb6e561bb7f3bd9d5d603
Parents: e9d835e
Author: Sam Whittle 
Authored: Mon Dec 5 14:57:29 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 5 20:54:19 2016 -0800

--
 .../beam/sdk/util/state/InMemoryTimerInternals.java   |  8 +---
 .../org/apache/beam/sdk/util/state/TimerCallback.java |  2 +-
 .../sdk/util/state/InMemoryTimerInternalsTest.java| 14 ++
 3 files changed, 20 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3434e8a4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
index 60c4a96..159b583 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
@@ -50,7 +50,7 @@ public class InMemoryTimerInternals implements TimerInternals 
{
   private PriorityQueue synchronizedProcessingTimers = new 
PriorityQueue<>(11);
 
   /** Current input watermark. */
-  @Nullable private Instant inputWatermarkTime = 
BoundedWindow.TIMESTAMP_MIN_VALUE;
+  private Instant inputWatermarkTime = BoundedWindow.TIMESTAMP_MIN_VALUE;
 
   /** Current output watermark. */
   @Nullable private Instant outputWatermarkTime = null;
@@ -59,7 +59,7 @@ public class InMemoryTimerInternals implements TimerInternals 
{
   private Instant processingTime = BoundedWindow.TIMESTAMP_MIN_VALUE;
 
   /** Current synchronized processing time. */
-  @Nullable private Instant synchronizedProcessingTime = null;
+  private Instant synchronizedProcessingTime = 
BoundedWindow.TIMESTAMP_MIN_VALUE;
 
   @Override
   @Nullable
@@ -142,7 +142,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 
   @Override
   public Instant currentInputWatermarkTime() {
-return checkNotNull(inputWatermarkTime);
+return inputWatermarkTime;
   }
 
   @Override
@@ -197,6 +197,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 
   /** Advances processing time to the given value. */
   public void advanceProcessingTime(Instant newProcessingTime) throws 
Exception {
+checkNotNull(newProcessingTime);
 checkState(
 !newProcessingTime.isBefore(processingTime),
 "Cannot move processing time backwards from %s to %s",
@@ -211,6 +212,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
   /** Advances synchronized processing time to the given value. */
   public void advanceSynchronizedProcessingTime(Instant 
newSynchronizedProcessingTime)
   throws Exception {
+checkNotNull(newSynchronizedProcessingTime);
 checkState(
 !newSynchronizedProcessingTime.isBefore(synchronizedProcessingTime),
 "Cannot move processing time backwards from %s to %s",

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3434e8a4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
index 1d68e36..83791d6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
@@ -22,7 +22,7 @@ import org.apache.beam.sdk.util.TimerInternals;
 /**
  * A callback that processes a {@link TimerInternals.TimerData TimerData}.
  *
- * @deprecated Use TimerInternals.advanceTime and removeTimers instead of 
callback.
+ * @deprecated Use InMemoryTimerInternals advance and remove methods instead 
of callback.
  */
 @Deprecated
 public interface TimerCallback {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3434e8a4/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
--
diff --git

[GitHub] incubator-beam pull request #1494: Fix auth related unit test failures

2016-12-05 Thread vikkyrk 
Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1494


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1507: merge master into gearpump-runner branch

2016-12-05 Thread manuzhang 
GitHub user manuzhang reopened a pull request:

https://github.com/apache/incubator-beam/pull/1507

merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1507


commit 73d0af9151340d85df1f720e88366f8a463b44bc
Author: Thomas Groh 
Date:   2016-11-23T00:14:29Z

Add TransformHierarchyTest

This tests basic features of TransformHierarchy

commit 3ae9425b3a36e34890980647952c61cedbd91ff3
Author: Kenneth Knowles 
Date:   2016-11-23T06:24:37Z

This closes #1425

commit 09986e9433d49812d5061fe6543dff90d78eba6a
Author: Kenneth Knowles 
Date:   2016-11-23T06:16:29Z

Use more natural class to find class loader in ReflectHelpers

commit 5f3e7787d7e724b827af8924b2773ed3b5c2b036
Author: Luke Cwik 
Date:   2016-11-23T15:42:13Z

Use more natural class to find class loader in ReflectHelpers

This closes #1427

commit 2e2146b1869807d69658592de8ed5ff339c28507
Author: Thomas Weise 
Date:   2016-11-22T19:38:00Z

Update transitive dependencies for Apex 3.5.0 snapshot version.

commit 26a30a22d28e034d16b5f0c4ea0be1b4f8c464f6
Author: Kenneth Knowles 
Date:   2016-11-23T17:56:17Z

This closes #1418

commit ef74e192eaee79e4cb8c7c901a296dd76559d76d
Author: Daniel Kulp 
Date:   2016-11-22T18:31:19Z

[BEAM-1034] Clean up tmp area in tests

commit 6d0c205a306d6cdca346fe2aaf662b03b4959a0e
Author: Jean-Baptiste Onofré 
Date:   2016-11-24T07:43:09Z

[BEAM-1034] This closes #1415

commit 7b314aad1c7c62ad61e09e610c60f53ac056d75d
Author: Jean-Baptiste Onofré 
Date:   2016-11-17T16:07:21Z

[BEAM-959] Improve validation messages in JdbcIO

commit 3e4b2fd0d96ff2757de7782b7c80dc1881eb451b
Author: Jean-Baptiste Onofré 
Date:   2016-11-24T09:31:52Z

[BEAM-959] This closes #1374

commit 4a097729ac9fc65283f4f11f85812188589c8df3
Author: Aljoscha Krettek 
Date:   2016-11-08T10:03:21Z

Replace WindowAssignment OldDoFn by FlatMap in Flink Runner

The streaming runner had an OldDoFn that was used for assigning windows
using a WindowFn. This is now done with a FlatMap.

commit 8d1214a3ba94b21102b74d346e73f24ecd9056f2
Author: Aljoscha Krettek 
Date:   2016-11-24T14:20:49Z

This closes #1435

commit 8d7d46c6e407c738a61b236078d002d178da0b9f
Author: manuzhang 
Date:   2016-11-23T01:24:05Z

[BEAM-800] add getFn to DoFnInvoker

commit 632576b5be00f050ff86981bfe55b170dec41759
Author: Kenneth Knowles 
Date:   2016-11-26T04:34:12Z

This closes #1428

commit 07544ef3a47bbdfacc00c75af875c3533a5fe477
Author: Kenneth Knowles 
Date:   2016-11-23T19:22:08Z

Remove unused body of StreamingPCollectionViewWriterFn

commit 803bbe2a3026424f509e13809a8eecb79990e5fe
Author: Kenneth Knowles 
Date:   2016-11-23T19:23:07Z

Remove unused WindowingInternals.writePCollectionViewData

commit 3ad7677503977108b5a67c315bd1cc6ead3ee998
Author: Sela 
Date:   2016-11-26T10:50:01Z

[BEAM-498] Remove obsolete WindowingInternals#writePCollectionViewData
This closes #1430

commit cc96b1381b6db849adf69daddecf30b9c61acf73
Author: Ismaël Mejía 
Date:   2016-11-25T13:52:26Z

[BEAM-851] Determine if the pipeline must be translated into streaming mode 
(if not set)

Now an Evaluator (visitor) detects if there are Unbonded.Read transforms.
This approach is based on Flink's PipelineTranslationOptimizer

commit 8cc43aa701807009ec826e752b2f1bb95442450f
Author: Sela 
Date:   2016-11-27T11:19:59Z

This closes #1436

commit f6005593ccf6bdef7c975622d5af39792c7db44c
Author: Ismaël Mejía 
Date:   2016-11-27T10:39:08Z

[BEAM-1049] Update spark version to

[jira] [Commented] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724042#comment-15724042
 ] 

ASF GitHub Bot commented on BEAM-964:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1514


> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1514: [BEAM-964] Rollback avro change

2016-12-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1514


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Rollback the default format to json

2016-12-05 Thread davor 
Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk eb98d636e -> e73bdb5c2


Rollback the default format to json


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b7748cca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b7748cca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b7748cca

Branch: refs/heads/python-sdk
Commit: b7748cca9c2da97337ae94c18c8b8b70e59ed655
Parents: eb98d63
Author: Sourabh Bajaj 
Authored: Mon Dec 5 18:03:43 2016 -0800
Committer: Sourabh Bajaj 
Committed: Mon Dec 5 18:03:43 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b7748cca/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index a3f7d94..8b953b0 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -520,7 +520,7 @@ class DataflowPipelineRunner(PipelineRunner):
 elif transform.source.format == 'text':
   step.add_property(PropertyNames.FILE_PATTERN, transform.source.path)
 elif transform.source.format == 'bigquery':
-  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_AVRO')
+  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_JSON')
   # TODO(silviuc): Add table validation if transform.source.validate.
   if transform.source.table_reference is not None:
 step.add_property(PropertyNames.BIGQUERY_DATASET,

[2/2] incubator-beam git commit: This closes #1514

2016-12-05 Thread davor 
This closes #1514


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e73bdb5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e73bdb5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e73bdb5c

Branch: refs/heads/python-sdk
Commit: e73bdb5c2f896c20663d6200400a1c34add1d7b8
Parents: eb98d63 b7748cc
Author: Davor Bonaci 
Authored: Mon Dec 5 18:04:46 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 5 18:04:46 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #1514: Rollback for release

2016-12-05 Thread sb2nov 
GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1514

Rollback for release

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @chamikaramj @robertwb PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam BEAM_rollback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1514


commit 4cb59130d87cf1c98c552f813b69ee717b1bd11a
Author: Sourabh Bajaj 
Date:   2016-12-06T01:53:19Z

Rollback for release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1513: [BEAM-551] Add support for BigQueryIO.Wri...

2016-12-05 Thread sammcveety 
GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1513

[BEAM-551] Add support for BigQueryIO.Write

R: @davorbonaci 

Based on https://github.com/apache/incubator-beam/pull/1238

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1513


commit e06fd8ce238971d1b235ce01f527f7cedb058127
Author: sammcveety 
Date:   2016-10-20T16:29:04Z

Add NestedValueProvider

commit 2e2aa465295dd141577639098522fdbae359126a
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

Add BQ VP code

commit 6fed9071be77b6b12506ca223a8fb68032887f5e
Author: Sam McVeety 
Date:   2016-10-31T17:39:10Z

BQ VP

commit b8974403bae0bf50da841f526f01085d99f89f97
Author: Sam McVeety 
Date:   2016-11-01T23:35:59Z

Merge branch 'master' into sgmc/bq_vp

commit 61b220f30b0e9fd67f688de7df50fd8909c58a01
Author: Sam McVeety 
Date:   2016-11-01T23:57:39Z

Fix most tests

commit 053267a41ffe78180c0bcef8046258314f8970d9
Author: Sam McVeety 
Date:   2016-11-29T18:16:25Z

Make serializable

commit 73d248e3e3fdf0d1b15abf60a1763c41e2054940
Author: Sam McVeety 
Date:   2016-12-01T21:39:39Z

Fix BQ tests

commit 407ee12e8f454889bc335bf17bb9c37dc0df33e5
Author: Sam McVeety 
Date:   2016-12-02T23:12:56Z

Merge branch 'master' into sgmc/bq_vp

commit 4958f8634298e1078a4e20b920dc158e630d6633
Author: Sam McVeety 
Date:   2016-12-02T23:23:48Z

Fix tests

commit 9a44144c475e19afa33103c3247672c71f7034be
Author: Sam McVeety 
Date:   2016-12-03T01:29:40Z

Update API

commit 9c13496df5e4630476925032ce596797471fd396
Author: Sam McVeety 
Date:   2016-12-03T02:11:19Z

BQ Write

commit aacb0281fe714e94658001746c97edf37049dd55
Author: Sam McVeety 
Date:   2016-12-04T22:16:23Z

Add schema parameterization

commit 99fc371a34ec24e6ee59f852db9a6d327bf5589b
Author: Sam McVeety 
Date:   2016-12-05T23:12:14Z

update bq

commit a5230670b3f893a0a97ee4eba434eb3bd7ca9ff0
Author: Sam McVeety 
Date:   2016-12-05T23:38:04Z

Fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723978#comment-15723978
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1513

[BEAM-551] Add support for BigQueryIO.Write

R: @davorbonaci 

Based on https://github.com/apache/incubator-beam/pull/1238

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1513


commit e06fd8ce238971d1b235ce01f527f7cedb058127
Author: sammcveety 
Date:   2016-10-20T16:29:04Z

Add NestedValueProvider

commit 2e2aa465295dd141577639098522fdbae359126a
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

Add BQ VP code

commit 6fed9071be77b6b12506ca223a8fb68032887f5e
Author: Sam McVeety 
Date:   2016-10-31T17:39:10Z

BQ VP

commit b8974403bae0bf50da841f526f01085d99f89f97
Author: Sam McVeety 
Date:   2016-11-01T23:35:59Z

Merge branch 'master' into sgmc/bq_vp

commit 61b220f30b0e9fd67f688de7df50fd8909c58a01
Author: Sam McVeety 
Date:   2016-11-01T23:57:39Z

Fix most tests

commit 053267a41ffe78180c0bcef8046258314f8970d9
Author: Sam McVeety 
Date:   2016-11-29T18:16:25Z

Make serializable

commit 73d248e3e3fdf0d1b15abf60a1763c41e2054940
Author: Sam McVeety 
Date:   2016-12-01T21:39:39Z

Fix BQ tests

commit 407ee12e8f454889bc335bf17bb9c37dc0df33e5
Author: Sam McVeety 
Date:   2016-12-02T23:12:56Z

Merge branch 'master' into sgmc/bq_vp

commit 4958f8634298e1078a4e20b920dc158e630d6633
Author: Sam McVeety 
Date:   2016-12-02T23:23:48Z

Fix tests

commit 9a44144c475e19afa33103c3247672c71f7034be
Author: Sam McVeety 
Date:   2016-12-03T01:29:40Z

Update API

commit 9c13496df5e4630476925032ce596797471fd396
Author: Sam McVeety 
Date:   2016-12-03T02:11:19Z

BQ Write

commit aacb0281fe714e94658001746c97edf37049dd55
Author: Sam McVeety 
Date:   2016-12-04T22:16:23Z

Add schema parameterization

commit 99fc371a34ec24e6ee59f852db9a6d327bf5589b
Author: Sam McVeety 
Date:   2016-12-05T23:12:14Z

update bq

commit a5230670b3f893a0a97ee4eba434eb3bd7ca9ff0
Author: Sam McVeety 
Date:   2016-12-05T23:38:04Z

Fixes




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

2016-12-05 Thread Sourabh Bajaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-964.

Resolution: Fixed

> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-852) Validate sources when they are created

2016-12-05 Thread Sourabh Bajaj (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723945#comment-15723945
 ] 

Sourabh Bajaj commented on BEAM-852:


[~chamikara] this should be fixed now.

> Validate sources when they are created
> --
>
> Key: BEAM-852
> URL: https://issues.apache.org/jira/browse/BEAM-852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> We currently do not validate some sources at creation time. For example text, 
> Avro. Validating sources early will improve user experience since it will  
> help catch issues early. For example, we can fail before submitting a job to 
> a runner.
> It should also be possible to disable validation to support environments 
> where users do not have access to the input at job submission. Java SDK 
> already follows a similar model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1731

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1731

2016-12-05 Thread Apache Jenkins Server 
See

[jira] [Commented] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723933#comment-15723933
 ] 

ASF GitHub Bot commented on BEAM-964:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1510


> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1510: [BEAM-964] Change export format to AVRO f...

2016-12-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1510


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Change export format to AVRO for BQ

2016-12-05 Thread robertwb 
Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk d59bccd82 -> eb98d636e


Change export format to AVRO for BQ


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/390fbfdd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/390fbfdd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/390fbfdd

Branch: refs/heads/python-sdk
Commit: 390fbfddf444df410f7cb61329496f6f24a0532c
Parents: d59bccd
Author: Sourabh Bajaj 
Authored: Mon Dec 5 14:04:54 2016 -0800
Committer: Sourabh Bajaj 
Committed: Mon Dec 5 14:04:54 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/390fbfdd/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 8b953b0..a3f7d94 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -520,7 +520,7 @@ class DataflowPipelineRunner(PipelineRunner):
 elif transform.source.format == 'text':
   step.add_property(PropertyNames.FILE_PATTERN, transform.source.path)
 elif transform.source.format == 'bigquery':
-  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_JSON')
+  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_AVRO')
   # TODO(silviuc): Add table validation if transform.source.validate.
   if transform.source.table_reference is not None:
 step.add_property(PropertyNames.BIGQUERY_DATASET,

[2/2] incubator-beam git commit: Closes #1510

2016-12-05 Thread robertwb 
Closes #1510


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eb98d636
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eb98d636
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/eb98d636

Branch: refs/heads/python-sdk
Commit: eb98d636ec9bbe11795aa9ee2fea1a8ceddaf794
Parents: d59bccd 390fbfd
Author: Robert Bradshaw 
Authored: Mon Dec 5 17:10:34 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Dec 5 17:10:34 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Updated] (BEAM-886) Support new DoFn in Python SDK

2016-12-05 Thread Ahmet Altay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-886:
-
Assignee: Sourabh Bajaj  (was: Ahmet Altay)

> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large

2016-12-05 Thread Neelesh Srinivas Salian (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723857#comment-15723857
 ] 

Neelesh Srinivas Salian commented on BEAM-1089:
---

[~kenn], I wanted to add that PR verbosity on the JIRA itself can be reduced. 
Apache Spark has this setup well where just the PR link shows up on the JIRA 
rather than the pr bot's messages.

> Jenkins comments on PRs are too many & too large
> 
>
> Key: BEAM-1089
> URL: https://issues.apache.org/jira/browse/BEAM-1089
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Lately, I've been finding review comments somewhat drowned out by asfbot 
> copying build results onto a PR. It also generates a lot of needless email. I 
> have not yet tried to devise just the right filter, hoping we can just return 
> to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1730

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1730

2016-12-05 Thread Apache Jenkins Server 
See

[jira] [Created] (BEAM-1091) E2E WordCount test in Precommit

2016-12-05 Thread Mark Liu (JIRA) 
Mark Liu created BEAM-1091:
--

 Summary: E2E WordCount test in Precommit
 Key: BEAM-1091
 URL: https://issues.apache.org/jira/browse/BEAM-1091
 Project: Beam
  Issue Type: Task
  Components: sdk-py, testing
Reporter: Mark Liu
Assignee: Mark Liu


We want to include some e2e test in precommit in order to catch bugs earlier, 
instead of breaking postcommit very often.

As what we have in postcommit, we want the same wordcount test in precommit 
executed by DirectPipelineRunner and DataflowRunner. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1090) High memory usage error

2016-12-05 Thread JIRA 
María GH created BEAM-1090:
--

 Summary: High memory usage error
 Key: BEAM-1090
 URL: https://issues.apache.org/jira/browse/BEAM-1090
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Affects Versions: 0.3.0-incubating
Reporter: María GH
Priority: Minor


Non-reproducible high memory usage test failure. It goes away on its own.
RuntimeError: High memory usage: 201418866688 > 201008464768 [while running 
'oom:check']
root: WARNING: A task failed with exception.
 High memory usage: 201418866688 > 201008464768 [while running 'oom:check']
---
Complete results at https://travis-ci.org/apache/incubator-beam/jobs/181011669



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1055) Display Data keys on Python are inconsistent

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723770#comment-15723770
 ] 

ASF GitHub Bot commented on BEAM-1055:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/incubator-beam/pull/1443

[BEAM-1055] Display Data keys on Python are inconsistent

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-1000smth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1443


commit d58d4cafa0a7fef9de59ee473d1262612bcf8694
Author: Pablo 
Date:   2016-11-28T21:05:47Z

Giving all display data keys in Python a standard snake_case format.




> Display Data keys on Python are inconsistent
> 
>
> Key: BEAM-1055
> URL: https://issues.apache.org/jira/browse/BEAM-1055
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>
> Some are in camelCase, some are in snake_case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1055) Display Data keys on Python are inconsistent

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723769#comment-15723769
 ] 

ASF GitHub Bot commented on BEAM-1055:
--

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1443


> Display Data keys on Python are inconsistent
> 
>
> Key: BEAM-1055
> URL: https://issues.apache.org/jira/browse/BEAM-1055
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>
> Some are in camelCase, some are in snake_case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1443: [BEAM-1055] Display Data keys on Python a...

2016-12-05 Thread pabloem 
GitHub user pabloem reopened a pull request:

https://github.com/apache/incubator-beam/pull/1443

[BEAM-1055] Display Data keys on Python are inconsistent

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-1000smth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1443


commit d58d4cafa0a7fef9de59ee473d1262612bcf8694
Author: Pablo 
Date:   2016-11-28T21:05:47Z

Giving all display data keys in Python a standard snake_case format.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1443: [BEAM-1055] Display Data keys on Python a...

2016-12-05 Thread pabloem 
Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1443


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1512: Move template_runners_test to runners fol...

2016-12-05 Thread aaltay 
GitHub user aaltay reopened a pull request:

https://github.com/apache/incubator-beam/pull/1512

Move template_runners_test to runners folder.

also removed the logging code, test does not use or import `logging`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam temp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1512


commit 540ddcca3ad4fe5fbf217072880be66122f3f3b4
Author: Ahmet Altay 
Date:   2016-12-05T22:57:14Z

Move template_runners_test to runners folder.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #1993

2016-12-05 Thread Apache Jenkins Server 
See

[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large

2016-12-05 Thread Neelesh Srinivas Salian (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723702#comment-15723702
 ] 

Neelesh Srinivas Salian commented on BEAM-1089:
---

+1 the failure messages clog up the PR completely.

> Jenkins comments on PRs are too many & too large
> 
>
> Key: BEAM-1089
> URL: https://issues.apache.org/jira/browse/BEAM-1089
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Lately, I've been finding review comments somewhat drowned out by asfbot 
> copying build results onto a PR. It also generates a lot of needless email. I 
> have not yet tried to devise just the right filter, hoping we can just return 
> to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #313

2016-12-05 Thread Apache Jenkins Server 
See

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #313

2016-12-05 Thread Apache Jenkins Server 
See

[jira] [Created] (BEAM-1089) Jenkins comments on PRs are too many & too large

2016-12-05 Thread Kenneth Knowles (JIRA) 
Kenneth Knowles created BEAM-1089:
-

 Summary: Jenkins comments on PRs are too many & too large
 Key: BEAM-1089
 URL: https://issues.apache.org/jira/browse/BEAM-1089
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Kenneth Knowles
Assignee: Jason Kuster


Lately, I've been finding review comments somewhat drowned out by asfbot 
copying build results onto a PR. It also generates a lot of needless email. I 
have not yet tried to devise just the right filter, hoping we can just return 
to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Flink #1002

2016-12-05 Thread Apache Jenkins Server 
See

[GitHub] incubator-beam pull request #1512: Move template_runners_test to runners fol...

2016-12-05 Thread aaltay 
GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1512

Move template_runners_test to runners folder.

also removed the logging code, test does not use or import `logging`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam temp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1512


commit 540ddcca3ad4fe5fbf217072880be66122f3f3b4
Author: Ahmet Altay 
Date:   2016-12-05T22:57:14Z

Move template_runners_test to runners folder.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1478: [BEAM-362] Moved KeyedWorkItem and relate...

2016-12-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1478


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: Update Dataflow worker to beam-master-20161205

2016-12-05 Thread kenn 
Update Dataflow worker to beam-master-20161205


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/81d12959
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/81d12959
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/81d12959

Branch: refs/heads/master
Commit: 81d129597e1f3393346564f083a62dc905d3869c
Parents: 6893a72
Author: Kenneth Knowles <k...@google.com>
Authored: Mon Dec 5 13:32:52 2016 -0800
Committer: Kenneth Knowles <k...@google.com>
Committed: Mon Dec 5 14:55:02 2016 -0800

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/81d12959/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 0357b46..f485cb8 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -210,9 +210,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161129";
+  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161205";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161129";
+  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161205";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723640#comment-15723640
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1478


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: This closes #1478

2016-12-05 Thread kenn 
Repository: incubator-beam
Updated Branches:
  refs/heads/master 6893a7270 -> b8e93fff8


This closes #1478


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b8e93fff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b8e93fff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b8e93fff

Branch: refs/heads/master
Commit: b8e93fff8d8ef85357eb5d4ade0a29dae60c3e04
Parents: 6893a72 8a2f020
Author: Kenneth Knowles 
Authored: Mon Dec 5 14:55:02 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 5 14:55:02 2016 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |   4 +-
 .../apache/beam/runners/core/DoFnRunner.java|   1 -
 .../apache/beam/runners/core/DoFnRunners.java   |   1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   1 -
 .../apache/beam/runners/core/KeyedWorkItem.java |  44 +++
 .../beam/runners/core/KeyedWorkItemCoder.java   | 130 +++
 .../beam/runners/core/KeyedWorkItems.java   | 122 +
 .../core/LateDataDroppingDoFnRunner.java|   2 -
 .../beam/runners/core/SplittableParDo.java  |   2 -
 .../runners/core/KeyedWorkItemCoderTest.java|  64 +
 .../beam/runners/core/SplittableParDoTest.java  |   2 -
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |   2 +-
 .../beam/runners/direct/DirectGroupByKey.java   |   4 +-
 .../direct/ExecutorServiceParallelExecutor.java |   4 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 +-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   4 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +-
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |   4 +-
 .../GroupByKeyOnlyEvaluatorFactoryTest.java |   4 +-
 .../streaming/SingletonKeyedWorkItem.java   |   2 +-
 .../streaming/SingletonKeyedWorkItemCoder.java  |   4 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   6 +-
 .../wrappers/streaming/WorkItemKeySelector.java |   2 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   4 +-
 .../org/apache/beam/sdk/util/KeyedWorkItem.java |  43 --
 .../beam/sdk/util/KeyedWorkItemCoder.java   | 128 --
 .../apache/beam/sdk/util/KeyedWorkItems.java| 121 -
 .../beam/sdk/util/KeyedWorkItemCoderTest.java   |  62 -
 28 files changed, 383 insertions(+), 388 deletions(-)
--

[jira] [Commented] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723510#comment-15723510
 ] 

ASF GitHub Bot commented on BEAM-964:
-

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1510

[BEAM-964] Change export format to AVRO for BQ

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-json-avro-bq-dataflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1510


commit 390fbfddf444df410f7cb61329496f6f24a0532c
Author: Sourabh Bajaj 
Date:   2016-12-05T22:04:54Z

Change export format to AVRO for BQ




> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1510: [BEAM-964] Change export format to AVRO f...

2016-12-05 Thread sb2nov 
GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1510

[BEAM-964] Change export format to AVRO for BQ

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-json-avro-bq-dataflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1510


commit 390fbfddf444df410f7cb61329496f6f24a0532c
Author: Sourabh Bajaj 
Date:   2016-12-05T22:04:54Z

Change export format to AVRO for BQ




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1064) Convert Jenkins jobs to DSL

2016-12-05 Thread Jason Kuster (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723493#comment-15723493
 ] 

Jason Kuster commented on BEAM-1064:


Everything done and committed except for Coveralls. Problem is that the 
currently-installed version of the credentials plugin doesn't mask the 
credentials in the log, which is a blocker. According to their release log it 
should be in the latest version; need to check which version is currently 
installed on our jenkins.

> Convert Jenkins jobs to DSL
> ---
>
> Key: BEAM-1064
> URL: https://issues.apache.org/jira/browse/BEAM-1064
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>  Labels: jenkins
>
> Move Jenkins jobs to DSL. PR is here:
> https://github.com/apache/incubator-beam/pull/1390



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1088) submit_job_description needs job argument (Fails post commit)

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723360#comment-15723360
 ] 

ASF GitHub Bot commented on BEAM-1088:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1509


> submit_job_description needs job argument (Fails post commit)
> -
>
> Key: BEAM-1088
> URL: https://issues.apache.org/jira/browse/BEAM-1088
> Project: Beam
>  Issue Type: Bug
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/822/consoleFull
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
>  line 106, in 
> run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
>  line 97, in run
> result = p.run()
>   File "apache_beam/pipeline.py", line 159, in run
> return self.runner.run(self)
>   File "apache_beam/runners/dataflow_runner.py", line 179, in run
> self.dataflow_client.create_job(self.job))
>   File "apache_beam/utils/retry.py", line 167, in wrapper
> return fun(*args, **kwargs)
>   File "apache_beam/internal/apiclient.py", line 415, in create_job
> return self.submit_job_description()
>   File "apache_beam/internal/apiclient.py", line 433, in 
> submit_job_description
> request.job = job.proto
> NameError: global name 'job' is not defined



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #825

2016-12-05 Thread Apache Jenkins Server 
See

Build failed in Jenkins: beam_PostCommit_Python_Verify #824

2016-12-05 Thread Apache Jenkins Server 
See 

--
[...truncated 3365 lines...]
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "s15", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
},

[2/2] incubator-beam git commit: Closes #1509

2016-12-05 Thread robertwb 
Closes #1509


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d59bccd8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d59bccd8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d59bccd8

Branch: refs/heads/python-sdk
Commit: d59bccd82461d340613a16ab41db2a4cc6e4200b
Parents: f7118c8 f1b83f7
Author: Robert Bradshaw 
Authored: Mon Dec 5 13:01:19 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Dec 5 13:01:19 2016 -0800

--
 sdks/python/apache_beam/internal/apiclient.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: Add missing job parameter to the submit_job_description.

2016-12-05 Thread robertwb 
Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk f7118c8a5 -> d59bccd82


Add missing job parameter to the submit_job_description.

Tested post commit test locally.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f1b83f7e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f1b83f7e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f1b83f7e

Branch: refs/heads/python-sdk
Commit: f1b83f7e82b56cd36bffbcdd5cc8ab319bf1e9d3
Parents: f7118c8
Author: Ahmet Altay 
Authored: Mon Dec 5 12:29:45 2016 -0800
Committer: Ahmet Altay 
Committed: Mon Dec 5 12:29:45 2016 -0800

--
 sdks/python/apache_beam/internal/apiclient.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f1b83f7e/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index a894557..c5f5f70 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -412,7 +412,7 @@ class DataflowApplicationClient(object):
   self.stage_file(gcs_or_local_path, file_name, StringIO(job.json()))
 
 if not template_location:
-  return self.submit_job_description()
+  return self.submit_job_description(job)
 else:
   return None
 
@@ -426,7 +426,7 @@ class DataflowApplicationClient(object):
 # TODO(silviuc): Remove the debug logging eventually.
 logging.info('JOB: %s', job)
 
-  def submit_job_description(self):
+  def submit_job_description(self, job):
 """Creates and excutes a job request."""
 request = dataflow.DataflowProjectsJobsCreateRequest()
 request.projectId = self.google_cloud_options.project

[jira] [Commented] (BEAM-1088) submit_job_description needs job argument (Fails post commit)

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723299#comment-15723299
 ] 

ASF GitHub Bot commented on BEAM-1088:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1509

[BEAM-1088] Add missing job parameter to the submit_job_description.

Tested post commit test locally.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam apic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1509


commit f1b83f7e82b56cd36bffbcdd5cc8ab319bf1e9d3
Author: Ahmet Altay 
Date:   2016-12-05T20:29:45Z

Add missing job parameter to the submit_job_description.

Tested post commit test locally.




> submit_job_description needs job argument (Fails post commit)
> -
>
> Key: BEAM-1088
> URL: https://issues.apache.org/jira/browse/BEAM-1088
> Project: Beam
>  Issue Type: Bug
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/822/consoleFull
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
>  line 106, in 
> run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
>  line 97, in run
> result = p.run()
>   File "apache_beam/pipeline.py", line 159, in run
> return self.runner.run(self)
>   File "apache_beam/runners/dataflow_runner.py", line 179, in run
> self.dataflow_client.create_job(self.job))
>   File "apache_beam/utils/retry.py", line 167, in wrapper
> return fun(*args, **kwargs)
>   File "apache_beam/internal/apiclient.py", line 415, in create_job
> return self.submit_job_description()
>   File "apache_beam/internal/apiclient.py", line 433, in 
> submit_job_description
> request.job = job.proto
> NameError: global name 'job' is not defined



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1509: [BEAM-1088] Add missing job parameter to ...

2016-12-05 Thread aaltay 
GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1509

[BEAM-1088] Add missing job parameter to the submit_job_description.

Tested post commit test locally.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam apic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1509


commit f1b83f7e82b56cd36bffbcdd5cc8ab319bf1e9d3
Author: Ahmet Altay 
Date:   2016-12-05T20:29:45Z

Add missing job parameter to the submit_job_description.

Tested post commit test locally.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1088) submit_job_description needs job argument (Fails post commit)

2016-12-05 Thread Ahmet Altay (JIRA) 
Ahmet Altay created BEAM-1088:
-

 Summary: submit_job_description needs job argument (Fails post 
commit)
 Key: BEAM-1088
 URL: https://issues.apache.org/jira/browse/BEAM-1088
 Project: Beam
  Issue Type: Bug
Reporter: Ahmet Altay
Assignee: Ahmet Altay


https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/822/consoleFull

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
 line 106, in 
run()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
 line 97, in run
result = p.run()
  File "apache_beam/pipeline.py", line 159, in run
return self.runner.run(self)
  File "apache_beam/runners/dataflow_runner.py", line 179, in run
self.dataflow_client.create_job(self.job))
  File "apache_beam/utils/retry.py", line 167, in wrapper
return fun(*args, **kwargs)
  File "apache_beam/internal/apiclient.py", line 415, in create_job
return self.submit_job_description()
  File "apache_beam/internal/apiclient.py", line 433, in submit_job_description
request.job = job.proto
NameError: global name 'job' is not defined



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #823

2016-12-05 Thread Apache Jenkins Server 
See 

Changes:

[vikasrk] Add experimental warning to datastoreio

--
[...truncated 3365 lines...]
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "s15", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",

[2/2] incubator-beam git commit: Add experimental warning to datastoreio

2016-12-05 Thread lcwik 
Add experimental warning to datastoreio

This closes #1508


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f7118c8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f7118c8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f7118c8a

Branch: refs/heads/python-sdk
Commit: f7118c8a592a27c8eedb1dced7cbf85a4084c390
Parents: 0d99856 f3dcf6c
Author: Luke Cwik 
Authored: Mon Dec 5 11:39:09 2016 -0800
Committer: Luke Cwik 
Committed: Mon Dec 5 11:39:09 2016 -0800

--
 sdks/python/apache_beam/io/datastore/v1/datastoreio.py | 2 ++
 1 file changed, 2 insertions(+)
--

[GitHub] incubator-beam pull request #1508: Add experimental warning to datastoreio

2016-12-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1508


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add experimental warning to datastoreio

2016-12-05 Thread lcwik 
Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 0d99856f3 -> f7118c8a5


Add experimental warning to datastoreio


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3dcf6c2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3dcf6c2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3dcf6c2

Branch: refs/heads/python-sdk
Commit: f3dcf6c2dd32e17998d9e185a326fb725abccf31
Parents: 0d99856
Author: Vikas Kedigehalli 
Authored: Mon Dec 5 11:23:53 2016 -0800
Committer: Vikas Kedigehalli 
Committed: Mon Dec 5 11:28:52 2016 -0800

--
 sdks/python/apache_beam/io/datastore/v1/datastoreio.py | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3dcf6c2/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
--
diff --git a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py 
b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
index 20466b9..fc3e813 100644
--- a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
@@ -86,6 +86,7 @@ class ReadFromDatastore(PTransform):
   namespace: An optional namespace.
   num_splits: Number of splits for the query.
 """
+logging.warning('datastoreio read transform is experimental.')
 super(ReadFromDatastore, self).__init__()
 
 if not project:
@@ -309,6 +310,7 @@ class _Mutate(PTransform):
  """
 self._project = project
 self._mutation_fn = mutation_fn
+logging.warning('datastoreio write transform is experimental.')
 
   def apply(self, pcoll):
 return (pcoll

[jira] [Updated] (BEAM-1087) Pickling error in save main session

2016-12-05 Thread Sourabh Bajaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-1087:

Description: 
{code}
  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
save_type
StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle :
 it's not found as 
apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum
{code}

  was:

  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
save_type
StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle :
 it's not found as 
apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum


> Pickling error in save main session
> ---
>
> Key: BEAM-1087
> URL: https://issues.apache.org/jira/browse/BEAM-1087
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> {code}
>   File "/usr/local/lib/python2.7/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
>   File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
> save_type
> StockPickler.save_global(pickler, obj)
>   File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
> (obj, module, name))
> pickle.PicklingError: Can't pickle  'apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum'>:
>  it's not found as 
> apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1087) Pickling error in save main session

2016-12-05 Thread Sourabh Bajaj (JIRA) 
Sourabh Bajaj created BEAM-1087:
---

 Summary: Pickling error in save main session
 Key: BEAM-1087
 URL: https://issues.apache.org/jira/browse/BEAM-1087
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Sourabh Bajaj
Assignee: Sourabh Bajaj
Priority: Minor


```
  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
save_type
StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle :
 it's not found as 
apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1087) Pickling error in save main session

2016-12-05 Thread Sourabh Bajaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-1087:

Description: 

  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
save_type
StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle :
 it's not found as 
apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum

  was:
```
  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
save_type
StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle :
 it's not found as 
apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum```


> Pickling error in save main session
> ---
>
> Key: BEAM-1087
> URL: https://issues.apache.org/jira/browse/BEAM-1087
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> 
>   File "/usr/local/lib/python2.7/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
>   File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
> save_type
> StockPickler.save_global(pickler, obj)
>   File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
> (obj, module, name))
> pickle.PicklingError: Can't pickle  'apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum'>:
>  it's not found as 
> apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1508: Add experimental warning to datastoreio

2016-12-05 Thread vikkyrk 
GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1508

Add experimental warning to datastoreio

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_exp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1508


commit b7f63b0840f3e14bf9be7c6833ea6eb878b84efc
Author: Vikas Kedigehalli 
Date:   2016-12-05T19:23:53Z

Add experimental warning to datastoreio




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723078#comment-15723078
 ] 

ASF GitHub Bot commented on BEAM-802:
-

Github user mariapython closed the pull request at:

https://github.com/apache/incubator-beam/pull/1342


> Support Dynamic PipelineOptions for python
> --
>
> Key: BEAM-802
> URL: https://issues.apache.org/jira/browse/BEAM-802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline 
> options)
> Background: Currently, the Runner creates the JSON pipeline description which 
> can be sent to the worker as is, since everything is already defined there 
> (with links to gs:// for input and binaries). With the parametrized approach, 
> those descriptions are empty and filled by the user or defaulted, so the 
> pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that 
> needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic 
> key/value pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1342: [BEAM-802] -- Add ability to stage a job ...

2016-12-05 Thread mariapython 
Github user mariapython closed the pull request at:

https://github.com/apache/incubator-beam/pull/1342


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #822

2016-12-05 Thread Apache Jenkins Server 
See 

Changes:

[robertwb] Modify create_job to allow staging the job and not submitting it to 
the

--
[...truncated 3365 lines...]
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "s15", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type":

[1/2] incubator-beam git commit: Modify create_job to allow staging the job and not submitting it to the service.

2016-12-05 Thread robertwb 
Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 7c5e4aa66 -> 0d99856f3


Modify create_job to allow staging the job and not submitting it to the service.

- Modularize create_job in create job description, stage job, and send for 
execution.
- Modify --dataflow_job_file to stage the job and continue submitting it to the 
service.
- Add --template_location to stage the job but not submit it to the service.
- Add tests for both, including making them mutually exclusive (following Java 
SDK decision).
- Add template_runner_test.py with integration tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cfa0ad81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cfa0ad81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cfa0ad81

Branch: refs/heads/python-sdk
Commit: cfa0ad8136b323bade9de14ea6149e7f74cbd0b4
Parents: 7c5e4aa
Author: Maria Garcia Herrero 
Authored: Wed Nov 2 09:14:48 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Dec 5 11:04:34 2016 -0800

--
 sdks/python/apache_beam/examples/wordcount.py   |  1 -
 sdks/python/apache_beam/internal/apiclient.py   | 34 +++-
 .../apache_beam/runners/dataflow_runner.py  | 13 ++-
 sdks/python/apache_beam/template_runner_test.py | 83 
 sdks/python/apache_beam/utils/options.py| 10 +++
 .../apache_beam/utils/pipeline_options_test.py  | 13 +++
 .../utils/pipeline_options_validator_test.py| 28 +++
 7 files changed, 175 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfa0ad81/sdks/python/apache_beam/examples/wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/wordcount.py 
b/sdks/python/apache_beam/examples/wordcount.py
index 096e508..7f347d8 100644
--- a/sdks/python/apache_beam/examples/wordcount.py
+++ b/sdks/python/apache_beam/examples/wordcount.py
@@ -59,7 +59,6 @@ class WordExtractingDoFn(beam.DoFn):
 
 def run(argv=None):
   """Main entry point; defines and runs the wordcount pipeline."""
-
   parser = argparse.ArgumentParser()
   parser.add_argument('--input',
   dest='input',

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfa0ad81/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index 5612631..a894557 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -24,6 +24,7 @@ import os
 import re
 import time
 
+from StringIO import StringIO
 
 from apitools.base.py import encoding
 from apitools.base.py import exceptions
@@ -42,7 +43,6 @@ from apache_beam.utils.options import DebugOptions
 from apache_beam.utils.options import GoogleCloudOptions
 from apache_beam.utils.options import StandardOptions
 from apache_beam.utils.options import WorkerOptions
-
 from apache_beam.internal.clients import storage
 import apache_beam.internal.clients.dataflow as dataflow
 
@@ -327,6 +327,9 @@ class Job(object):
 self.base64_str_re = re.compile(r'^[A-Za-z0-9+/]*=*$')
 self.coder_str_re = re.compile(r'^([A-Za-z]+\$)([A-Za-z0-9+/]*=*)$')
 
+  def json(self):
+return encoding.MessageToJson(self.proto)
+
 
 class DataflowApplicationClient(object):
   """A Dataflow API client used by application code to create and query 
jobs."""
@@ -392,8 +395,29 @@ class DataflowApplicationClient(object):
   # TODO(silviuc): Refactor so that retry logic can be applied.
   @retry.no_retries  # Using no_retries marks this as an integration point.
   def create_job(self, job):
-"""Submits for remote execution a job described by the workflow proto."""
-# Stage job resources and add an environment proto with their paths.
+"""Creates a job description.
+Additionally, it may stage it and/or submit it for remote execution.
+"""
+self.create_job_description(job)
+
+# Stage and submit the job when necessary
+dataflow_job_file = job.options.view_as(DebugOptions).dataflow_job_file
+template_location = (
+job.options.view_as(GoogleCloudOptions).template_location)
+
+job_location = template_location or dataflow_job_file
+if job_location:
+  gcs_or_local_path = os.path.dirname(job_location)
+  file_name = os.path.basename(job_location)
+  self.stage_file(gcs_or_local_path, file_name, StringIO(job.json()))
+
+if not template_location:
+  return self.submit_job_description()
+else:
+  return None
+
+  def create_job_description(self, job):
+"""Creates a job described by the

[2/2] incubator-beam git commit: Closes #1342

2016-12-05 Thread robertwb 
Closes #1342


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0d99856f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0d99856f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0d99856f

Branch: refs/heads/python-sdk
Commit: 0d99856f37d6bca9bb8d676ae36157bd0515a4f2
Parents: 7c5e4aa cfa0ad8
Author: Robert Bradshaw 
Authored: Mon Dec 5 11:04:35 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Dec 5 11:04:35 2016 -0800

--
 sdks/python/apache_beam/examples/wordcount.py   |  1 -
 sdks/python/apache_beam/internal/apiclient.py   | 34 +++-
 .../apache_beam/runners/dataflow_runner.py  | 13 ++-
 sdks/python/apache_beam/template_runner_test.py | 83 
 sdks/python/apache_beam/utils/options.py| 10 +++
 .../apache_beam/utils/pipeline_options_test.py  | 13 +++
 .../utils/pipeline_options_validator_test.py| 28 +++
 7 files changed, 175 insertions(+), 7 deletions(-)
--

[jira] [Commented] (BEAM-1046) Travis (Linux) failing for python sdk

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723007#comment-15723007
 ] 

ASF GitHub Bot commented on BEAM-1046:
--

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1456


> Travis (Linux) failing for python sdk
> -
>
> Key: BEAM-1046
> URL: https://issues.apache.org/jira/browse/BEAM-1046
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>
> All PRs are failing with the same error:
> An example: https://travis-ci.org/apache/incubator-beam/builds/178435675
> $ if [ "$TEST_PYTHON" ] && ! pip list | grep tox; then travis_retry pip 
> install tox --user `whoami`; fi
> You are using pip version 6.0.8, however version 9.0.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Collecting tox
>   Downloading tox-2.5.0-py2.py3-none-any.whl (42kB)
> 100% || 45kB 6.3MB/s 
> Collecting travis
>   Could not find any downloads that satisfy the requirement travis
>   No distributions at all found for travis
> The command "pip install tox --user travis" failed. Retrying, 2 of 3.
> You are using pip version 6.0.8, however version 9.0.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Collecting tox
>   Using cached tox-2.5.0-py2.py3-none-any.whl
> Collecting travis
>   Could not find any downloads that satisfy the requirement travis
>   No distributions at all found for travis
> The command "pip install tox --user travis" failed. Retrying, 3 of 3.
> You are using pip version 6.0.8, however version 9.0.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Collecting tox
>   Using cached tox-2.5.0-py2.py3-none-any.whl
> Collecting travis
>   Could not find any downloads that satisfy the requirement travis
>   No distributions at all found for travis
> The command "pip install tox --user travis" failed 3 times.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1456: [BEAM-1046] - Travis tests on Python fail...

2016-12-05 Thread pabloem 
Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1456


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722650#comment-15722650
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1459


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1459: [BEAM-551] Changes to BigQuery validation...

2016-12-05 Thread sammcveety 
Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-165) Add Hadoop MapReduce runner

2016-12-05 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/BEAM-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-165:
--
Summary: Add Hadoop MapReduce runner  (was: Add a MapReduce runner)

> Add Hadoop MapReduce runner
> ---
>
> Key: BEAM-165
> URL: https://issues.apache.org/jira/browse/BEAM-165
> Project: Beam
>  Issue Type: Wish
>  Components: runner-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> I think a MapReduce runner could be a good addition to Beam. It would allow 
> users to smoothly "migrate" from MapReduce to Spark or Flink.
> Of course, the MapReduce runner will run in batch mode (not stream).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-425) Create Elasticsearch IO

2016-12-05 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/BEAM-425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-425:
--
Summary: Create Elasticsearch IO  (was: Add Elasticsearch IO)

> Create Elasticsearch IO
> ---
>
> Key: BEAM-425
> URL: https://issues.apache.org/jira/browse/BEAM-425
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> I'm working on a new ElasticsearchIO providing both bounded source and sink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-214) Create Parquet IO

2016-12-05 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/BEAM-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-214:
--
Summary: Create Parquet IO  (was: Create ParquetIO)

> Create Parquet IO
> -
>
> Key: BEAM-214
> URL: https://issues.apache.org/jira/browse/BEAM-214
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Neville Li
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Would be nice to support Parquet files with projection and predicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-214) Create ParquetIO

2016-12-05 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/BEAM-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-214:
--
Summary: Create ParquetIO  (was: Parquet IO)

> Create ParquetIO
> 
>
> Key: BEAM-214
> URL: https://issues.apache.org/jira/browse/BEAM-214
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Neville Li
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Would be nice to support Parquet files with projection and predicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1050) PipelineResult.State is not set to FAILED when a streaming job fails

2016-12-05 Thread Amit Sela (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1050.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> PipelineResult.State is not set to FAILED when a streaming job fails
> 
>
> Key: BEAM-1050
> URL: https://issues.apache.org/jira/browse/BEAM-1050
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> In case of failure, {{SteamingContext#awaitTerminationOrTimeout}} and 
> {{SteamingContext#awaitTermination}} will both throw an exception, and so 
> {{state = State.DONE}} will not be executed in the code block below. 
> In addition, it would probably make sense to set {{state = State.FAILED}} in 
> cases where an exception takes place.
> {code:java}
> if (isStreamingPipeline()) {
>   // stop streaming context
>   if (timeout > 0) {
> jssc.awaitTerminationOrTimeout(timeout);
>   } else {
> jssc.awaitTermination();
>   }
>   // stop streaming context gracefully, so checkpointing (and other 
> computations) get to
>   // finish before shutdown.
>   jssc.stop(false, gracefully);
> }
> state = State.DONE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

2016-12-05 Thread Amit Sela (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-595.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() 
> in Spark EvaluationContext
> ---
>
> Key: BEAM-595
> URL: https://issues.apache.org/jira/browse/BEAM-595
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Pei He
> Fix For: 0.4.0-incubating
>
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationExcedption in Spark runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

2016-12-05 Thread Amit Sela (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-595:
---
Assignee: (was: Kobi Salant)

> Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() 
> in Spark EvaluationContext
> ---
>
> Key: BEAM-595
> URL: https://issues.apache.org/jira/browse/BEAM-595
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Pei He
> Fix For: 0.4.0-incubating
>
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationExcedption in Spark runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1000) Add non blocking cancel() and waitUntilFinish() for Spark batch application

2016-12-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722029#comment-15722029
 ] 

ASF GitHub Bot commented on BEAM-1000:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1466


> Add non blocking cancel() and waitUntilFinish() for Spark batch application
> ---
>
> Key: BEAM-1000
> URL: https://issues.apache.org/jira/browse/BEAM-1000
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Kobi Salant
>Assignee: Stas Levin
> Fix For: 0.4.0-incubating
>
>
> Spark batch submit by nature is blocking. 
> To enable cancel and waitForFinish we will need to run it in a different 
> thread/future and return a context which communicates with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1000) Add non blocking cancel() and waitUntilFinish() for Spark batch application

2016-12-05 Thread Amit Sela (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1000.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Add non blocking cancel() and waitUntilFinish() for Spark batch application
> ---
>
> Key: BEAM-1000
> URL: https://issues.apache.org/jira/browse/BEAM-1000
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Kobi Salant
>Assignee: Stas Levin
> Fix For: 0.4.0-incubating
>
>
> Spark batch submit by nature is blocking. 
> To enable cancel and waitForFinish we will need to run it in a different 
> thread/future and return a context which communicates with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: This closes #1466

2016-12-05 Thread amitsela 
This closes #1466


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6893a727
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6893a727
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6893a727

Branch: refs/heads/master
Commit: 6893a7270b728ec72c6e8749008e6a583edf5921
Parents: ef9871c 158378f
Author: Sela 
Authored: Mon Dec 5 12:57:25 2016 +0200
Committer: Sela 
Committed: Mon Dec 5 12:57:25 2016 +0200

--
 .../beam/runners/spark/EvaluationResult.java|  67 --
 .../beam/runners/spark/SparkPipelineResult.java | 193 +
 .../apache/beam/runners/spark/SparkRunner.java  | 113 ++
 .../beam/runners/spark/TestSparkRunner.java |  11 +-
 .../spark/aggregators/AccumulatorSingleton.java |   6 +-
 .../spark/aggregators/SparkAggregators.java |  97 +
 .../beam/runners/spark/examples/WordCount.java  |   2 +-
 .../spark/translation/EvaluationContext.java| 131 ++-
 .../spark/translation/SparkContextFactory.java  |   2 +-
 .../spark/translation/SparkRuntimeContext.java  |  62 +-
 .../spark/translation/TransformTranslator.java  |  10 +-
 .../streaming/StreamingTransformTranslator.java |  10 +-
 .../runners/spark/ProvidedSparkContextTest.java |   6 +-
 .../runners/spark/SparkPipelineStateTest.java   | 217 +++
 .../spark/aggregators/ClearAggregatorsRule.java |  37 
 .../metrics/sink/ClearAggregatorsRule.java  |  33 ---
 .../metrics/sink/NamedAggregatorsTest.java  |   3 +-
 .../beam/runners/spark/io/AvroPipelineTest.java |   2 +-
 .../beam/runners/spark/io/NumShardsTest.java|   2 +-
 .../io/hadoop/HadoopFileFormatPipelineTest.java |   2 +-
 .../spark/translation/SideEffectsTest.java  |  59 -
 .../streaming/EmptyStreamAssertionTest.java |   4 +
 .../ResumeFromCheckpointStreamingTest.java  |  15 +-
 .../streaming/utils/PAssertStreaming.java   |   9 +-
 24 files changed, 680 insertions(+), 413 deletions(-)
--

[1/3] incubator-beam git commit: [BEAM-1000, BEAM-1050] Fixed PipelineResult.State Failed for streaming, support non-blocking cancel/waituntilfinish in batch. Added a SparkPipelineResult class to addr

2016-12-05 Thread amitsela 
Repository: incubator-beam
Updated Branches:
  refs/heads/master ef9871c36 -> 6893a7270


[BEAM-1000, BEAM-1050] Fixed PipelineResult.State Failed for streaming, support 
non-blocking
cancel/waituntilfinish in batch.
Added a SparkPipelineResult class to address PipelineResult#waitUntilFinish() 
and SparkRunner#run() semantics.

* Simplified beamExceptionFrom() to abstract away SparkExceptions.
* Reordered methods according to access level.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b1a67934
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b1a67934
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b1a67934

Branch: refs/heads/master
Commit: b1a67934d1496e221718599301635c38f8e3b7ec
Parents: ef9871c
Author: Stas Levin 
Authored: Mon Nov 28 11:11:10 2016 +0200
Committer: Sela 
Committed: Mon Dec 5 12:56:39 2016 +0200

--
 .../beam/runners/spark/EvaluationResult.java|  67 --
 .../beam/runners/spark/SparkPipelineResult.java | 179 +++
 .../apache/beam/runners/spark/SparkRunner.java  |  98 +
 .../beam/runners/spark/TestSparkRunner.java |  10 +-
 .../beam/runners/spark/examples/WordCount.java  |   2 +-
 .../spark/translation/EvaluationContext.java| 119 ++
 .../spark/translation/SparkContextFactory.java  |   2 +-
 .../runners/spark/ProvidedSparkContextTest.java |   6 +-
 .../runners/spark/SparkPipelineStateTest.java   | 219 +++
 .../metrics/sink/ClearAggregatorsRule.java  |   2 +-
 .../metrics/sink/NamedAggregatorsTest.java  |   2 +-
 .../beam/runners/spark/io/AvroPipelineTest.java |   2 +-
 .../beam/runners/spark/io/NumShardsTest.java|   2 +-
 .../io/hadoop/HadoopFileFormatPipelineTest.java |   2 +-
 .../spark/translation/SideEffectsTest.java  |  59 -
 .../streaming/EmptyStreamAssertionTest.java |   4 +
 .../ResumeFromCheckpointStreamingTest.java  |   8 +-
 .../streaming/utils/PAssertStreaming.java   |   9 +-
 18 files changed, 500 insertions(+), 292 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b1a67934/runners/spark/src/main/java/org/apache/beam/runners/spark/EvaluationResult.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/EvaluationResult.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/EvaluationResult.java
deleted file mode 100644
index 52606a3..000
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/EvaluationResult.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.runners.spark;
-
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PValue;
-
-/**
- * Interface for retrieving the result(s) of running a pipeline. Allows us to 
translate between
- * {@code PObject}s or {@code PCollection}s and Ts or collections of Ts.
- */
-public interface EvaluationResult extends PipelineResult {
-  /**
-   * Retrieves an iterable of results associated with the PCollection passed 
in.
-   *
-   * @param pcollection Collection we wish to translate.
-   * @param  Type of elements contained in collection.
-   * @return Natively types result associated with collection.
-   */
-   Iterable get(PCollection pcollection);
-
-  /**
-   * Retrieve an object of Type T associated with the PValue passed in.
-   *
-   * @param pval PValue to retrieve associated data for.
-   * @param   Type of object to return.
-   * @return Native object.
-   */
-   T get(PValue pval);
-
-  /**
-   * Retrieves the final value of the aggregator.
-   *
-   * @param aggNamename of aggregator.
-   * @param resultType Class of final result of aggregation.
-   * @param Type of final result of aggregation.
-   * @return Result of aggregation associated with specified name.
-   */
-   T

[GitHub] incubator-beam pull request #1466: [BEAM-1000,BEAM-595,BEAM-1050] Aligned th...

2016-12-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1466


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Redistributed some responsibilities in order to remove getAggregatorValues() form EvaluationContext.

2016-12-05 Thread amitsela 
Redistributed some responsibilities in order to remove getAggregatorValues() 
form EvaluationContext.

Inferred excepted exception handling according to existing codebase and tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/158378f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/158378f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/158378f0

Branch: refs/heads/master
Commit: 158378f0f682b80462b917002b895ddbf782d06d
Parents: b1a6793
Author: Stas Levin 
Authored: Sat Dec 3 00:47:39 2016 +0200
Committer: Sela 
Committed: Mon Dec 5 12:56:41 2016 +0200

--
 .../beam/runners/spark/SparkPipelineResult.java | 76 ---
 .../apache/beam/runners/spark/SparkRunner.java  | 35 +--
 .../beam/runners/spark/TestSparkRunner.java |  1 +
 .../spark/aggregators/AccumulatorSingleton.java |  6 +-
 .../spark/aggregators/SparkAggregators.java | 97 
 .../spark/translation/EvaluationContext.java| 20 +---
 .../spark/translation/SparkRuntimeContext.java  | 62 +
 .../spark/translation/TransformTranslator.java  | 10 +-
 .../streaming/StreamingTransformTranslator.java | 10 +-
 .../runners/spark/SparkPipelineStateTest.java   | 36 
 .../spark/aggregators/ClearAggregatorsRule.java | 37 
 .../metrics/sink/ClearAggregatorsRule.java  | 33 ---
 .../metrics/sink/NamedAggregatorsTest.java  |  1 +
 .../streaming/EmptyStreamAssertionTest.java |  2 +-
 .../ResumeFromCheckpointStreamingTest.java  |  9 +-
 15 files changed, 247 insertions(+), 188 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/158378f0/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
index ec0610c..b1027a6 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
@@ -23,7 +23,7 @@ import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
-import org.apache.beam.runners.spark.translation.EvaluationContext;
+import org.apache.beam.runners.spark.aggregators.SparkAggregators;
 import org.apache.beam.runners.spark.translation.SparkContextFactory;
 import org.apache.beam.sdk.AggregatorRetrievalException;
 import org.apache.beam.sdk.AggregatorValues;
@@ -31,7 +31,10 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.spark.SparkException;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
 import org.joda.time.Duration;
 
 /**
@@ -40,29 +43,37 @@ import org.joda.time.Duration;
 public abstract class SparkPipelineResult implements PipelineResult {
 
   protected final Future pipelineExecution;
-  protected final EvaluationContext context;
+  protected JavaSparkContext javaSparkContext;
 
   protected PipelineResult.State state;
 
   SparkPipelineResult(final Future pipelineExecution,
-  final EvaluationContext evaluationContext) {
+  final JavaSparkContext javaSparkContext) {
 this.pipelineExecution = pipelineExecution;
-this.context = evaluationContext;
+this.javaSparkContext = javaSparkContext;
 // pipelineExecution is expected to have started executing eagerly.
 state = State.RUNNING;
   }
 
-  private RuntimeException runtimeExceptionFrom(Throwable e) {
+  private RuntimeException runtimeExceptionFrom(final Throwable e) {
 return (e instanceof RuntimeException) ? (RuntimeException) e : new 
RuntimeException(e);
   }
 
-  private RuntimeException beamExceptionFrom(Throwable e) {
+  private RuntimeException beamExceptionFrom(final Throwable e) {
 // Scala doesn't declare checked exceptions in the bytecode, and the Java 
compiler
 // won't let you catch something that is not declared, so we can't catch
 // SparkException directly, instead we do an instanceof check.
-return (e instanceof SparkException)
-? new Pipeline.PipelineExecutionException(e.getCause() != null ? 
e.getCause() : e)
-: runtimeExceptionFrom(e);
+
+if (e instanceof SparkException) {
+  if (e.getCause() !=

[GitHub] incubator-beam pull request #1507: merge master into gearpump-runner branch

2016-12-05 Thread manuzhang 
GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1507

merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1507


commit 73d0af9151340d85df1f720e88366f8a463b44bc
Author: Thomas Groh 
Date:   2016-11-23T00:14:29Z

Add TransformHierarchyTest

This tests basic features of TransformHierarchy

commit 3ae9425b3a36e34890980647952c61cedbd91ff3
Author: Kenneth Knowles 
Date:   2016-11-23T06:24:37Z

This closes #1425

commit 09986e9433d49812d5061fe6543dff90d78eba6a
Author: Kenneth Knowles 
Date:   2016-11-23T06:16:29Z

Use more natural class to find class loader in ReflectHelpers

commit 5f3e7787d7e724b827af8924b2773ed3b5c2b036
Author: Luke Cwik 
Date:   2016-11-23T15:42:13Z

Use more natural class to find class loader in ReflectHelpers

This closes #1427

commit 2e2146b1869807d69658592de8ed5ff339c28507
Author: Thomas Weise 
Date:   2016-11-22T19:38:00Z

Update transitive dependencies for Apex 3.5.0 snapshot version.

commit 26a30a22d28e034d16b5f0c4ea0be1b4f8c464f6
Author: Kenneth Knowles 
Date:   2016-11-23T17:56:17Z

This closes #1418

commit ef74e192eaee79e4cb8c7c901a296dd76559d76d
Author: Daniel Kulp 
Date:   2016-11-22T18:31:19Z

[BEAM-1034] Clean up tmp area in tests

commit 6d0c205a306d6cdca346fe2aaf662b03b4959a0e
Author: Jean-Baptiste Onofré 
Date:   2016-11-24T07:43:09Z

[BEAM-1034] This closes #1415

commit 7b314aad1c7c62ad61e09e610c60f53ac056d75d
Author: Jean-Baptiste Onofré 
Date:   2016-11-17T16:07:21Z

[BEAM-959] Improve validation messages in JdbcIO

commit 3e4b2fd0d96ff2757de7782b7c80dc1881eb451b
Author: Jean-Baptiste Onofré 
Date:   2016-11-24T09:31:52Z

[BEAM-959] This closes #1374

commit 4a097729ac9fc65283f4f11f85812188589c8df3
Author: Aljoscha Krettek 
Date:   2016-11-08T10:03:21Z

Replace WindowAssignment OldDoFn by FlatMap in Flink Runner

The streaming runner had an OldDoFn that was used for assigning windows
using a WindowFn. This is now done with a FlatMap.

commit 8d1214a3ba94b21102b74d346e73f24ecd9056f2
Author: Aljoscha Krettek 
Date:   2016-11-24T14:20:49Z

This closes #1435

commit 8d7d46c6e407c738a61b236078d002d178da0b9f
Author: manuzhang 
Date:   2016-11-23T01:24:05Z

[BEAM-800] add getFn to DoFnInvoker

commit 632576b5be00f050ff86981bfe55b170dec41759
Author: Kenneth Knowles 
Date:   2016-11-26T04:34:12Z

This closes #1428

commit 07544ef3a47bbdfacc00c75af875c3533a5fe477
Author: Kenneth Knowles 
Date:   2016-11-23T19:22:08Z

Remove unused body of StreamingPCollectionViewWriterFn

commit 803bbe2a3026424f509e13809a8eecb79990e5fe
Author: Kenneth Knowles 
Date:   2016-11-23T19:23:07Z

Remove unused WindowingInternals.writePCollectionViewData

commit 3ad7677503977108b5a67c315bd1cc6ead3ee998
Author: Sela 
Date:   2016-11-26T10:50:01Z

[BEAM-498] Remove obsolete WindowingInternals#writePCollectionViewData
This closes #1430

commit cc96b1381b6db849adf69daddecf30b9c61acf73
Author: Ismaël Mejía 
Date:   2016-11-25T13:52:26Z

[BEAM-851] Determine if the pipeline must be translated into streaming mode 
(if not set)

Now an Evaluator (visitor) detects if there are Unbonded.Read transforms.
This approach is based on Flink's PipelineTranslationOptimizer

commit 8cc43aa701807009ec826e752b2f1bb95442450f
Author: Sela 
Date:   2016-11-27T11:19:59Z

This closes #1436

commit f6005593ccf6bdef7c975622d5af39792c7db44c
Author: Ismaël Mejía 
Date:   2016-11-27T10:39:08Z

[BEAM-1049] Update spark version to