Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #10

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-380) Remove Spark runner dependency on beam-examlpes-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390492#comment-15390492
 ] 

ASF GitHub Bot commented on BEAM-380:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/539


> Remove Spark runner dependency on beam-examlpes-java
> 
>
> Key: BEAM-380
> URL: https://issues.apache.org/jira/browse/BEAM-380
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>
> Remove runner dependency to allow to allow beam-examples to have runtime 
> dependency on runners-spark. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #539: [BEAM-380] Remove Spark runner dependency ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/539


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Remove JavaDoc links to beam-examples.

[dhalperi]

Remove JavaDoc links to beam-examples.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d3dc3686
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d3dc3686
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d3dc3686

Branch: refs/heads/master
Commit: d3dc368666d5b700c5026fe08455e1210692ce3e
Parents: 12fb778
Author: Sela 
Authored: Mon Jul 4 15:33:57 2016 +0300
Committer: Dan Halperin 
Committed: Fri Jul 22 19:51:50 2016 -0700

--
 .../java/org/apache/beam/runners/spark/examples/WordCount.java| 3 +--
 .../src/test/java/org/apache/beam/runners/spark/TfIdfTest.java| 3 +--
 2 files changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d3dc3686/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
index defd635..4951043 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
@@ -35,8 +35,7 @@ import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
 /**
- * Duplicated from {@link org.apache.beam.examples.WordCount} to avoid 
dependency on
- * beam-examlpes.
+ * Duplicated to avoid dependency on beam-examples.
  */
 public class WordCount {
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d3dc3686/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
index 4052e65..d1f8d12 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/TfIdfTest.java
@@ -77,8 +77,7 @@ public class TfIdfTest {
   }
 
   /**
-   * Duplicated from {@link org.apache.beam.examples.complete.TfIdf} to avoid 
dependency on
-   * beam-examlpes.
+   * Duplicated to avoid dependency on beam-examlpes.
*/
   public static class ComputeTfIdf
   extends PTransform>, PCollection

[3/3] incubator-beam git commit: Remove dependency on beam-examples-java.

[dhalperi]

Remove dependency on beam-examples-java.

Duplicate WordCount into spark examlpes package.

Duplicate parts of TfIdf from beam examlpes.

Better reuse of WordCount and its parts.

Remove dependency on beam-examples-java


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/12fb7781
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/12fb7781
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/12fb7781

Branch: refs/heads/master
Commit: 12fb77811d7bf8c72a3c0a48f391bab3c815fbef
Parents: 122cd04
Author: Sela 
Authored: Mon Jun 27 20:59:07 2016 +0300
Committer: Dan Halperin 
Committed: Fri Jul 22 19:51:50 2016 -0700

--
 runners/spark/pom.xml   |  11 -
 .../beam/runners/spark/examples/WordCount.java  | 138 +
 .../beam/runners/spark/SimpleWordCountTest.java |  76 +--
 .../apache/beam/runners/spark/TfIdfTest.java| 200 ++-
 .../beam/runners/spark/io/NumShardsTest.java|   2 +-
 .../translation/WindowedWordCountTest.java  |  12 +-
 .../streaming/SimpleStreamingWordCountTest.java |   6 +-
 7 files changed, 355 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/12fb7781/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 7f65e16..665f15d 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -222,17 +222,6 @@
   
 
 
-  org.apache.beam
-  beam-examples-java
-  
-
-
-  org.slf4j
-  slf4j-jdk14
-
-  
-
-
   org.apache.avro
   avro-mapred
   ${avro.version}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/12fb7781/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
new file mode 100644
index 000..defd635
--- /dev/null
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.examples;
+
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.Sum;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+/**
+ * Duplicated from {@link org.apache.beam.examples.WordCount} to avoid 
dependency on
+ * beam-examlpes.
+ */
+public class WordCount {
+
+  /**
+   * Concept #2: You can make your pipeline code less verbose by defining your 
DoFns statically out-
+   * of-line. This DoFn tokenizes lines of text into individual words; we pass 
it to a ParDo in the
+   * pipeline.
+   */
+  static class ExtractWordsFn extends DoFn {
+private final Aggregator emptyLines =
+createAggregator("emptyLines", new Sum.SumLongFn());
+
+@Override
+public void processElement(ProcessContext c) {
+  if (c.element().trim().isEmpty()) {
+emptyLines.addValue(1L);
+  }
+
+  // Split the line into words.
+  String[] words = c.element().split("[^a-zA-Z']+");
+
+  // 

[1/3] incubator-beam git commit: Closes #539

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 122cd0466 -> cf1464465


Closes #539


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cf146446
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cf146446
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cf146446

Branch: refs/heads/master
Commit: cf1464465e1e65be2e4896cc6f693f8bbe82b844
Parents: 122cd04 d3dc368
Author: Dan Halperin 
Authored: Fri Jul 22 19:51:50 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 19:51:50 2016 -0700

--
 runners/spark/pom.xml   |  11 -
 .../beam/runners/spark/examples/WordCount.java  | 137 +
 .../beam/runners/spark/SimpleWordCountTest.java |  76 +--
 .../apache/beam/runners/spark/TfIdfTest.java| 199 ++-
 .../beam/runners/spark/io/NumShardsTest.java|   2 +-
 .../translation/WindowedWordCountTest.java  |  12 +-
 .../streaming/SimpleStreamingWordCountTest.java |   6 +-
 7 files changed, 353 insertions(+), 90 deletions(-)
--

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390464#comment-15390464
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/714


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #714: [BEAM-79] Add Gearpump runner to runners p...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/714


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-484) Datastore Source should support Dynamic Splitting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390439#comment-15390439
 ] 

ASF GitHub Bot commented on BEAM-484:
-

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/722

[BEAM-484] Dynamic Rebalance support for Datastore Source

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Add support for Dynamic rebalance
- Move and rename DatastoreSource/Reader to V1Beta3Source/Reader into a new 
file.
- Test Read transform, Source and Reader independently.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam 
vikasrk/datastore_dyn_rebalance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/722.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #722


commit 8528a1a733e0d54682388411f8dffa620b1cd6a1
Author: Vikas Kedigehalli 
Date:   2016-07-19T18:51:05Z

Dynamic Rebalance support for Datastore Source




> Datastore Source should support Dynamic Splitting
> -
>
> Key: BEAM-484
> URL: https://issues.apache.org/jira/browse/BEAM-484
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #722: [BEAM-484] Dynamic Rebalance support for D...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/722

[BEAM-484] Dynamic Rebalance support for Datastore Source

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Add support for Dynamic rebalance
- Move and rename DatastoreSource/Reader to V1Beta3Source/Reader into a new 
file.
- Test Read transform, Source and Reader independently.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam 
vikasrk/datastore_dyn_rebalance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/722.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #722


commit 8528a1a733e0d54682388411f8dffa620b1cd6a1
Author: Vikas Kedigehalli 
Date:   2016-07-19T18:51:05Z

Dynamic Rebalance support for Datastore Source




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-484) Datastore Source should support Dynamic Splitting

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-484:
--

 Summary: Datastore Source should support Dynamic Splitting
 Key: BEAM-484
 URL: https://issues.apache.org/jira/browse/BEAM-484
 Project: Beam
  Issue Type: Improvement
Reporter: Vikas Kedigehalli
Assignee: Vikas Kedigehalli






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli resolved BEAM-354.

   Resolution: Fixed
Fix Version/s: 0.2.0-incubating

> Modify DatastoreIO to use Datastore v1beta3 API
> ---
>
> Key: BEAM-354
> URL: https://issues.apache.org/jira/browse/BEAM-354
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 0.2.0-incubating
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: 0.2.0-incubating
>
>
> Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the 
> DatastoreIO needs to be migrated to use the new version. Also in the process 
> of doing so, this is a good time to add a level of indirection via a 
> PTranform such that future changes in Datastore API would not result in 
> changing user/pipeline code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #9

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #705: Minor cdef value changes.

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/705


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #702: Add tests for WindowedValue.

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/702


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #721: Make DoFnRunner a Receiver.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/721

Make DoFnRunner a Receiver.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam dofn-receiver

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #721


commit 943adfd6327321f09d60ae4302aeb470aa57e558
Author: Robert Bradshaw 
Date:   2016-07-23T00:33:57Z

Make DoFnRunner a Receiver.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #713

[dhalperi]

Closes #713


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/47042ce8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/47042ce8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/47042ce8

Branch: refs/heads/python-sdk
Commit: 47042ce89cca22e48f00c3b9f8219eed85c119c6
Parents: 09b4daf 5bfd4cb
Author: Dan Halperin 
Authored: Fri Jul 22 17:17:53 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 17:17:53 2016 -0700

--
 sdks/python/apache_beam/io/gcsio.py | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--

[GitHub] incubator-beam pull request #713: Log all exceptions in _start_upload

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/713


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Log all exceptions in _start_upload

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 09b4daf23 -> 47042ce89


Log all exceptions in _start_upload


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5bfd4cb7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5bfd4cb7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5bfd4cb7

Branch: refs/heads/python-sdk
Commit: 5bfd4cb7e67dbe54cde728be0dc99a0343e64e69
Parents: 09b4daf
Author: Ahmet Altay 
Authored: Fri Jul 22 10:24:24 2016 -0700
Committer: Ahmet Altay 
Committed: Fri Jul 22 11:55:12 2016 -0700

--
 sdks/python/apache_beam/io/gcsio.py | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5bfd4cb7/sdks/python/apache_beam/io/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcsio.py 
b/sdks/python/apache_beam/io/gcsio.py
index 10409c9..9377266 100644
--- a/sdks/python/apache_beam/io/gcsio.py
+++ b/sdks/python/apache_beam/io/gcsio.py
@@ -579,11 +579,10 @@ class GcsBufferedWriter(object):
 # a time, buffering writes until that size is reached.
 try:
   self.client.objects.Insert(self.insert_request, upload=self.upload)
-except HttpError as http_error:
+except Exception as e:  # pylint: disable=broad-except
   logging.error(
-  'HTTP error while inserting file %s: %s', self.path, http_error)
-  self.upload_thread.last_error = http_error
-  raise
+  'Error in _start_upload while inserting file %s: %s', self.path, e)
+  self.upload_thread.last_error = e
 finally:
   self.child_conn.close()
 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #8

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #720: Remove overrides of isStreaming() and getA...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/720

Remove overrides of isStreaming() and getAppName() in SparkPipelineOptions

isStreaming() is already default to false implicitly, since it is not set.
getAppName() defaults to the name of the class that constructs the 
PipelineOptions via the PipelineOptionsFactory.

I think both should work in Spark. And, I see no reason to override them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam spark-app-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/720.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #720


commit 7bbfef844caf31c14e3d0255d3073bec1d987476
Author: Pei He 
Date:   2016-07-22T23:49:34Z

Remove overrides of isStreaming() and getAppName() in SparkPipelineOptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-480] Move insertAll() from BigQueryTableInserter to BigQueryServices

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 12b60ffa5 -> 122cd0466


[BEAM-480] Move insertAll() from BigQueryTableInserter to BigQueryServices


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bf3af5de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bf3af5de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bf3af5de

Branch: refs/heads/master
Commit: bf3af5de9e6f8126ffd2ccd3f7a68a84e55e90ff
Parents: 12b60ff
Author: Pei He 
Authored: Fri Jul 22 15:03:38 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 16:52:49 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  36 ++--
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  13 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 192 +++
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 192 ---
 .../gcp/bigquery/BigQueryServicesImplTest.java  |  70 +++
 .../gcp/bigquery/BigQueryTableInserterTest.java |  64 ---
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  40 +---
 7 files changed, 300 insertions(+), 307 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bf3af5de/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 130d444..76f7079 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -1668,11 +1668,13 @@ public class BigQueryIO {
   @Override
   public PDone apply(PCollection input) {
 BigQueryOptions options = 
input.getPipeline().getOptions().as(BigQueryOptions.class);
+BigQueryServices bqServices = getBigQueryServices();
 
 // In a streaming job, or when a tablespec function is defined, we use 
StreamWithDeDup
 // and BigQuery's streaming import API.
 if (options.isStreaming() || tableRefFunction != null) {
-  return input.apply(new StreamWithDeDup(getTable(), tableRefFunction, 
getSchema()));
+  return input.apply(
+  new StreamWithDeDup(getTable(), tableRefFunction, getSchema(), 
bqServices));
 }
 
 TableReference table = fromJsonString(jsonTableRef, 
TableReference.class);
@@ -1693,7 +1695,6 @@ public class BigQueryIO {
   e);
 }
 
-BigQueryServices bqServices = getBigQueryServices();
 return input.apply("Write", org.apache.beam.sdk.io.Write.to(
 new BigQuerySink(
 jobIdToken,
@@ -2018,6 +2019,8 @@ public class BigQueryIO {
 /** TableSchema in JSON. Use String to make the class Serializable. */
 private final String jsonTableSchema;
 
+private final BigQueryServices bqServices;
+
 /** JsonTableRows to accumulate BigQuery rows in order to batch writes. */
 private transient Map tableRows;
 
@@ -2034,8 +2037,9 @@ public class BigQueryIO {
 createAggregator("ByteCount", new Sum.SumLongFn());
 
 /** Constructor. */
-StreamingWriteFn(TableSchema schema) {
-  jsonTableSchema = toJsonString(schema);
+StreamingWriteFn(TableSchema schema, BigQueryServices bqServices) {
+  this.jsonTableSchema = toJsonString(schema);
+  this.bqServices = checkNotNull(bqServices, "bqServices");
 }
 
 /** Prepares a target BigQuery table. */
@@ -2060,11 +2064,10 @@ public class BigQueryIO {
 @Override
 public void finishBundle(Context context) throws Exception {
   BigQueryOptions options = 
context.getPipelineOptions().as(BigQueryOptions.class);
-  Bigquery client = Transport.newBigQueryClient(options).build();
 
   for (Map.Entry entry : tableRows.entrySet()) {
 TableReference tableReference = getOrCreateTable(options, 
entry.getKey());
-flushRows(client, tableReference, entry.getValue(),
+flushRows(tableReference, entry.getValue(),
 uniqueIdsForTableRows.get(entry.getKey()), options);
   }
   tableRows.clear();
@@ -2100,13 +2103,17 @@ public class BigQueryIO {
   return tableReference;
 }
 
-/** Writes the accumulated rows into BigQuery with streaming API. */
-private void flushRows(Bigquery client, TableReference tableReference,
-List tableRows, List uniqueIds, BigQueryOptions 
options) {
+/**
+ * Writes the accumulated rows into 

[GitHub] incubator-beam pull request #717: [BEAM-480] Move insertAll() from BigQueryT...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/717


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-480) Use BigQueryServices abstraction in BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390382#comment-15390382
 ] 

ASF GitHub Bot commented on BEAM-480:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/717


> Use BigQueryServices abstraction in BigQueryIO
> --
>
> Key: BEAM-480
> URL: https://issues.apache.org/jira/browse/BEAM-480
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> There are legacy code that sent request to BigQuery directly.
> They should be moved to use BigQueryServices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #717

[dhalperi]

Closes #717


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/122cd046
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/122cd046
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/122cd046

Branch: refs/heads/master
Commit: 122cd04663529f4fc44530ed9e144e0dfd68360f
Parents: 12b60ff bf3af5d
Author: Dan Halperin 
Authored: Fri Jul 22 16:52:50 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 16:52:50 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  36 ++--
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  13 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 192 +++
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 192 ---
 .../gcp/bigquery/BigQueryServicesImplTest.java  |  70 +++
 .../gcp/bigquery/BigQueryTableInserterTest.java |  64 ---
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  40 +---
 7 files changed, 300 insertions(+), 307 deletions(-)
--

[1/2] incubator-beam git commit: Added BigInteger to TypeDescriptors class.

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 07a313f1a -> 12b60ffa5


Added BigInteger to TypeDescriptors class.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/75df931a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/75df931a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/75df931a

Branch: refs/heads/master
Commit: 75df931a55dbd3f93d9143574cdad7291d3457aa
Parents: 07a313f
Author: Jesse Anderson 
Authored: Fri Jul 22 13:54:26 2016 -0700
Committer: Jesse Anderson 
Committed: Fri Jul 22 13:54:26 2016 -0700

--
 .../org/apache/beam/sdk/values/TypeDescriptors.java| 13 +
 1 file changed, 13 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/75df931a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
index b833c10..0e7b9cb 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.values;
 import com.google.common.reflect.TypeParameter;
 
 import java.math.BigDecimal;
+import java.math.BigInteger;
 import java.util.List;
 import java.util.Set;
 
@@ -115,6 +116,18 @@ public class TypeDescriptors {
   }
 
   /**
+   * The {@link TypeDescriptor} for BigInteger.
+   * This is the equivalent of:
+   * 
+   * new TypeDescriptorBigInteger() {};
+   * 
+   * @return A {@link TypeDescriptor} for BigInteger
+   */
+  public static TypeDescriptor bigintegers() {
+return new TypeDescriptor() {};
+  }
+
+  /**
* The {@link TypeDescriptor} for String.
* This is the equivalent of:
* 

[jira] [Commented] (BEAM-459) Add BigInteger to TypeDescriptors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390379#comment-15390379
 ] 

ASF GitHub Bot commented on BEAM-459:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/716


> Add BigInteger to TypeDescriptors
> -
>
> Key: BEAM-459
> URL: https://issues.apache.org/jira/browse/BEAM-459
> Project: Beam
>  Issue Type: Bug
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
>
> The TypeDescriptors class is missing a BigInteger TypeDescriptor method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #716: [BEAM-459] Add BigInteger to TypeDescripto...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/716


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #716

[dhalperi]

Closes #716


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/12b60ffa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/12b60ffa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/12b60ffa

Branch: refs/heads/master
Commit: 12b60ffa5d18812674299a97e84af31d063e170c
Parents: 07a313f 75df931
Author: Dan Halperin 
Authored: Fri Jul 22 16:51:37 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 16:51:37 2016 -0700

--
 .../org/apache/beam/sdk/values/TypeDescriptors.java| 13 +
 1 file changed, 13 insertions(+)
--

[jira] [Commented] (BEAM-483) Generated job names easily collide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390369#comment-15390369
 ] 

ASF GitHub Bot commented on BEAM-483:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/719

[BEAM-483] Moves NormalizedUniqueName to ApplicationNameOptions, and uses 
it in runners.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam job-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #719


commit e597582da820016f97709c847329237841a3ddd6
Author: Pei He 
Date:   2016-07-22T23:40:54Z

[BEAM-483] Move NormalizedUniqueName to ApplicationNameOptions




> Generated job names easily collide
> --
>
> Key: BEAM-483
> URL: https://issues.apache.org/jira/browse/BEAM-483
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> The current job name generation scheme may easily lead to duplicate job names 
> and cause DataflowJobAlreadyExistsException, especially when a series of jobs 
> are submitted at the same time. (e.g., from a single script).
> It would be better to just add a random suffix like "-3275" or "-x1bh".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #719: [BEAM-483] Moves NormalizedUniqueName to A...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/719

[BEAM-483] Moves NormalizedUniqueName to ApplicationNameOptions, and uses 
it in runners.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam job-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #719


commit e597582da820016f97709c847329237841a3ddd6
Author: Pei He 
Date:   2016-07-22T23:40:54Z

[BEAM-483] Move NormalizedUniqueName to ApplicationNameOptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-480) Use BigQueryServices abstraction in BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390312#comment-15390312
 ] 

ASF GitHub Bot commented on BEAM-480:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/717

[BEAM-480] Move insertAll() from BigQueryTableInserter to BigQueryServices



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam log-request

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #717


commit 2d998bd8dd99c2c03420a70fbed0194b2df95d23
Author: Pei He 
Date:   2016-07-22T22:03:38Z

[BEAM-480] Move insertAll() to BigQueryServices




> Use BigQueryServices abstraction in BigQueryIO
> --
>
> Key: BEAM-480
> URL: https://issues.apache.org/jira/browse/BEAM-480
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> There are legacy code that sent request to BigQuery directly.
> They should be moved to use BigQueryServices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #717: [BEAM-480] Move insertAll() from BigQueryT...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/717

[BEAM-480] Move insertAll() from BigQueryTableInserter to BigQueryServices



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam log-request

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #717


commit 2d998bd8dd99c2c03420a70fbed0194b2df95d23
Author: Pei He 
Date:   2016-07-22T22:03:38Z

[BEAM-480] Move insertAll() to BigQueryServices




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-482) Log RPC requests when they fails to improve error messages

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-482:

Component/s: sdk-java-gcp

> Log RPC requests when they fails to improve error messages
> --
>
> Key: BEAM-482
> URL: https://issues.apache.org/jira/browse/BEAM-482
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Priority: Minor
>
> For example, BigQuery sometime doesn't provide enough information in the 
> respond. Users can only know what is invalid in the request, if we log the it.
> Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
> 400 Bad Request
> {
>   "code" : 400,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Invalid value for: String is not a valid value",
> "reason" : "invalid"
>   } ],
>   "message" : "Invalid value for: String is not a valid value"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:427)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:416)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.getOrCreateTable(BigQueryTableInserter.java:373)
> at 
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.getOrCreateTable(BigQueryIO.java:2127)
> at 
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.finishBundle(BigQueryIO.java:2099)
> at 
> com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.finishBundle(DoFnRunnerBase.java:158)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-482) Log RPC requests when they fails to improve error messages

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He reassigned BEAM-482:
---

Assignee: Pei He

> Log RPC requests when they fails to improve error messages
> --
>
> Key: BEAM-482
> URL: https://issues.apache.org/jira/browse/BEAM-482
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> For example, BigQuery sometime doesn't provide enough information in the 
> respond. Users can only know what is invalid in the request, if we log the it.
> Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
> 400 Bad Request
> {
>   "code" : 400,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Invalid value for: String is not a valid value",
> "reason" : "invalid"
>   } ],
>   "message" : "Invalid value for: String is not a valid value"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:427)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:416)
> at 
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.getOrCreateTable(BigQueryTableInserter.java:373)
> at 
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.getOrCreateTable(BigQueryIO.java:2127)
> at 
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.finishBundle(BigQueryIO.java:2099)
> at 
> com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.finishBundle(DoFnRunnerBase.java:158)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-433) Make Beam examples runners agnostic

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-433:

Component/s: examples-java

> Make Beam examples runners agnostic
> ---
>
> Key: BEAM-433
> URL: https://issues.apache.org/jira/browse/BEAM-433
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
>
> Beam examples are ported from Dataflow, and they heavily reference to 
> Dataflow classes.
> There are following cleanup tasks:
> 1. Remove Dataflow streaming and batch injector setup (Done).
> 2. Remove references to DataflowPipelineOptions.
> 3. Move cancel() from DataflowPipelineJob to PipelineResult.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-481) Create a PubSubServices abstraction

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-481:

Component/s: sdk-java-gcp

> Create a PubSubServices abstraction
> ---
>
> Key: BEAM-481
> URL: https://issues.apache.org/jira/browse/BEAM-481
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Priority: Minor
>
> A PubSubServices interface (similar to BigQueryServices) can help to improve 
> testability and code quality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-483) Generated job names easily collide

[Pei He (JIRA)]

Pei He created BEAM-483:
---

 Summary: Generated job names easily collide
 Key: BEAM-483
 URL: https://issues.apache.org/jira/browse/BEAM-483
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Pei He
Assignee: Pei He
Priority: Minor


The current job name generation scheme may easily lead to duplicate job names 
and cause DataflowJobAlreadyExistsException, especially when a series of jobs 
are submitted at the same time. (e.g., from a single script).

It would be better to just add a random suffix like "-3275" or "-x1bh".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-481) Create a PubSubServices abstraction

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390268#comment-15390268
 ] 

Pei He commented on BEAM-481:
-

This will also help to address issue: 
https://issues.apache.org/jira/browse/BEAM-482
because the code interacts with GCP is centralized.

> Create a PubSubServices abstraction
> ---
>
> Key: BEAM-481
> URL: https://issues.apache.org/jira/browse/BEAM-481
> Project: Beam
>  Issue Type: Improvement
>Reporter: Pei He
>Priority: Minor
>
> A PubSubServices interface (similar to BigQueryServices) can help to improve 
> testability and code quality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-480) Use BigQueryServices abstraction in BigQueryIO

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390269#comment-15390269
 ] 

Pei He commented on BEAM-480:
-

This will also help to address issue: 
https://issues.apache.org/jira/browse/BEAM-482
because the code interacts with GCP is centralized.

> Use BigQueryServices abstraction in BigQueryIO
> --
>
> Key: BEAM-480
> URL: https://issues.apache.org/jira/browse/BEAM-480
> Project: Beam
>  Issue Type: Improvement
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> There are legacy code that sent request to BigQuery directly.
> They should be moved to use BigQueryServices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-481) Create a PubSubServices abstraction

[Pei He (JIRA)]

Pei He created BEAM-481:
---

 Summary: Create a PubSubServices abstraction
 Key: BEAM-481
 URL: https://issues.apache.org/jira/browse/BEAM-481
 Project: Beam
  Issue Type: Improvement
Reporter: Pei He
Priority: Minor


A PubSubServices interface (similar to BigQueryServices) can help to improve 
testability and code quality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-480) Use BigQueryServices abstraction in BigQueryIO

[Pei He (JIRA)]

Pei He created BEAM-480:
---

 Summary: Use BigQueryServices abstraction in BigQueryIO
 Key: BEAM-480
 URL: https://issues.apache.org/jira/browse/BEAM-480
 Project: Beam
  Issue Type: Improvement
Reporter: Pei He
Assignee: Pei He
Priority: Minor


There are legacy code that sent request to BigQuery directly.
They should be moved to use BigQueryServices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #716: [BEAM-459] Add BigInteger to TypeDescripto...

[eljefe6a]

GitHub user eljefe6a opened a pull request:

https://github.com/apache/incubator-beam/pull/716

[BEAM-459] Add BigInteger to TypeDescriptors

Added BigInteger to TypeDescriptors class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eljefe6a/incubator-beam BigIntegerTypeDesc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #716


commit 75df931a55dbd3f93d9143574cdad7291d3457aa
Author: Jesse Anderson 
Date:   2016-07-22T20:54:26Z

Added BigInteger to TypeDescriptors class.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-459) Add BigInteger to TypeDescriptors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390183#comment-15390183
 ] 

ASF GitHub Bot commented on BEAM-459:
-

GitHub user eljefe6a opened a pull request:

https://github.com/apache/incubator-beam/pull/716

[BEAM-459] Add BigInteger to TypeDescriptors

Added BigInteger to TypeDescriptors class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eljefe6a/incubator-beam BigIntegerTypeDesc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #716


commit 75df931a55dbd3f93d9143574cdad7291d3457aa
Author: Jesse Anderson 
Date:   2016-07-22T20:54:26Z

Added BigInteger to TypeDescriptors class.




> Add BigInteger to TypeDescriptors
> -
>
> Key: BEAM-459
> URL: https://issues.apache.org/jira/browse/BEAM-459
> Project: Beam
>  Issue Type: Bug
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
>
> The TypeDescriptors class is missing a BigInteger TypeDescriptor method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #715: Make save_main_session optional

[silviulica]

GitHub user silviulica opened a pull request:

https://github.com/apache/incubator-beam/pull/715

Make save_main_session optional

This PR does 3 things:
1) make --save_main_session optional
2) Sets it to true in all examples that need it (e.g., wordcount needs it)
3) refactor setup.py to separate as much as possible setup logic from 
strings/versions.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/silviulica/incubator-beam 
main_session_optional

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #715


commit e739431b7bbc64c39bc3384daaf4706972021bf0
Author: Silviu Calinoiu 
Date:   2016-07-22T20:14:14Z

Make save_main_session optional




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390102#comment-15390102
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/714

[BEAM-79] Add Gearpump runner to runners parent pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

I think this line got lost in the shuffle. It is why we aren't seeing the 
module in the Jenkins web UI, though the [tests are running and 
passing](https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/).

R: @dhalperi 
CC: @manuzhang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam gearpump-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714


commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c
Author: Kenneth Knowles 
Date:   2016-07-22T19:50:43Z

Add Gearpump runner to runners parent pom.xml




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #714: [BEAM-79] Add Gearpump runner to runners p...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/714

[BEAM-79] Add Gearpump runner to runners parent pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

I think this line got lost in the shuffle. It is why we aren't seeing the 
module in the Jenkins web UI, though the [tests are running and 
passing](https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/).

R: @dhalperi 
CC: @manuzhang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam gearpump-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714


commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c
Author: Kenneth Knowles 
Date:   2016-07-22T19:50:43Z

Add Gearpump runner to runners parent pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #713: Log all exceptions in _start_upload

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/713

Log all exceptions in _start_upload

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #713


commit 5bfd4cb7e67dbe54cde728be0dc99a0343e64e69
Author: Ahmet Altay 
Date:   2016-07-22T17:24:24Z

Log all exceptions in _start_upload




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #7

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #807

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #6

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #806

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #5

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-479) Move RunnableOnService test executions to postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389106#comment-15389106
 ] 

ASF GitHub Bot commented on BEAM-479:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/709


> Move RunnableOnService test executions to postcommit
> 
>
> Key: BEAM-479
> URL: https://issues.apache.org/jira/browse/BEAM-479
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-flink, runner-gearpump, 
> runner-spark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The Spark and Flink RunnableOnService tests only use a local endpoint, so 
> they _can_ run as unit tests, but our test duration is getting out of hand. 
> For Gearpump, the tests timeout. So this ticket tracks getting everyone to a 
> symmetric configuration.
> Later, we can re-enable a select few local endpoint tests for the various 
> runners, to smoke test, and hopefully get actual cluster-based integration 
> tests running on Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #709: [BEAM-479] Execute RunnableOnService tests...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/709


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #709

[dhalperi]

Closes #709


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/07a313f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/07a313f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/07a313f1

Branch: refs/heads/master
Commit: 07a313f1af78349040afd9ff6b01e60cdcd567a3
Parents: adb4720 060bc80
Author: Dan Halperin 
Authored: Fri Jul 22 00:39:59 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 00:39:59 2016 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 31 ---
 runners/pom.xml| 72 ++---
 2 files changed, 42 insertions(+), 61 deletions(-)
--

[1/2] incubator-beam git commit: Execute RunnableOnService tests only when runner options provided

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master adb472083 -> 07a313f1a


Execute RunnableOnService tests only when runner options provided

Previously, the situation was this:

 - All runners inherit a RunnableOnService integration-test
   execution referencing runnableOnServicePipelineOptions
   whether or not the variable was set. Basically an unbound
   variable reference.
 - The Dataflow runner had a profile disabling it if
   runnableOnServicePipelineOptions was not set.
 - Before they got configured, Flink and Spark had to
   do extra work to explicitly prevent the invalid
   configuration from being used.

After this change:

 - All runners inherit the same integration-test execution
   but only if the variable it requires is present.
 - Dataflow doesn't have any special profile.
 - Flink and Spark are unchanged, since they do set
   up the variable themselves. When they move to running
   only as postcommit, like Dataflow does, the hardcoding
   is expected to either move to a profile or move to
   the Jenkins invocation.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/060bc802
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/060bc802
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/060bc802

Branch: refs/heads/master
Commit: 060bc802f0bff8ca1a076e9f93a623058e1d651b
Parents: adb4720
Author: Kenneth Knowles 
Authored: Thu Jul 21 21:01:36 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jul 22 00:39:58 2016 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 31 ---
 runners/pom.xml| 72 ++---
 2 files changed, 42 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/060bc802/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 6a15635..f2c0fe8 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -37,37 +37,6 @@
 -MM-dd 
HH:mm
   
 
-  
-
-  
-  disable-runnable-on-service-tests
-  
-
-  !runnableOnServicePipelineOptions
-
-  
-  
-
-  
-org.apache.maven.plugins
-maven-surefire-plugin
-
-  
-runnable-on-service-tests
-
-  true
-
-  
-
-  
-
-  
-
-  
-
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/060bc802/runners/pom.xml
--
diff --git a/runners/pom.xml b/runners/pom.xml
index 612d493..777325d 100644
--- a/runners/pom.xml
+++ b/runners/pom.xml
@@ -39,34 +39,46 @@
 spark
   
 
-  
-
-  
-
-  org.apache.maven.plugins
-  maven-surefire-plugin
-  
-
-  runnable-on-service-tests
-  integration-test
-  
-test
-  
-  
-org.apache.beam.sdk.testing.RunnableOnService
-all
-4
-
-  org.apache.beam:beam-sdks-java-core
-
-
-  
${runnableOnServicePipelineOptions}
-
-  
-
-  
-
-  
-
-  
+  
+
+
+
+  runnable-on-service-tests
+  
+runnableOnServicePipelineOptions
+  
+  
+
+  
+
+  org.apache.maven.plugins
+  maven-surefire-plugin
+  
+
+  runnable-on-service-tests
+  integration-test
+  
+test
+  
+  
+
org.apache.beam.sdk.testing.RunnableOnService
+all
+4
+
+  
org.apache.beam:beam-sdks-java-core
+
+
+  
${runnableOnServicePipelineOptions}
+
+  
+
+  
+
+  
+
+  
+
+  
 

[GitHub] incubator-beam pull request #712: spark: Direct runner should be test scope

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/712

spark: Direct runner should be test scope



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam spark-test-dep

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/712.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #712


commit b4a2ddf332675a854277cc2990b5e59d9f761b5c
Author: Dan Halperin 
Date:   2016-07-22T07:37:07Z

spark: Direct runner should be test scope




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_PythonVerify #138

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #4

[Apache Jenkins Server]

See