Jenkins build is back to stable : beam_PostCommit_MavenVerify #1541

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1541

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-596) Support cancel() and waitUntilFinish() in DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577065#comment-15577065
 ] 

ASF GitHub Bot commented on BEAM-596:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1098


> Support cancel() and waitUntilFinish() in DirectRunner
> --
>
> Key: BEAM-596
> URL: https://issues.apache.org/jira/browse/BEAM-596
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Pei He
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1098: [BEAM-596] Replace DirectResult#awaitComp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1098


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: [BEAM-596] Replace DirectResult#awaitCompletion with waitUntilFinish

[lcwik]

[BEAM-596] Replace DirectResult#awaitCompletion with waitUntilFinish

This closes #1098


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b8e6eea6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b8e6eea6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b8e6eea6

Branch: refs/heads/master
Commit: b8e6eea691b48e14c4e2c3e84609d750769e09ee
Parents: c2c650a 06bd074
Author: Luke Cwik 
Authored: Fri Oct 14 17:21:46 2016 -0700
Committer: Luke Cwik 
Committed: Fri Oct 14 17:21:46 2016 -0700

--
 .../beam/runners/direct/CompletionCallback.java |  4 +-
 .../beam/runners/direct/DirectOptions.java  |  4 +-
 .../beam/runners/direct/DirectRunner.java   | 27 +++--
 .../direct/ExecutorServiceParallelExecutor.java | 40 ++--
 .../beam/runners/direct/PipelineExecutor.java   |  2 +-
 .../beam/runners/direct/TransformExecutor.java  | 10 ++---
 .../beam/runners/direct/DirectRunnerTest.java   |  6 +--
 .../runners/direct/TransformExecutorTest.java   | 16 
 8 files changed, 54 insertions(+), 55 deletions(-)
--

[2/3] incubator-beam git commit: Stop Catching Errors in the DirectRunner

[lcwik]

Stop Catching Errors in the DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5e51c840
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5e51c840
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5e51c840

Branch: refs/heads/master
Commit: 5e51c84003c2c9e03d51f94cbc2be07569bf090e
Parents: c2c650a
Author: Thomas Groh 
Authored: Fri Oct 14 10:32:14 2016 -0700
Committer: Luke Cwik 
Committed: Fri Oct 14 17:21:18 2016 -0700

--
 .../beam/runners/direct/CompletionCallback.java |  4 +-
 .../beam/runners/direct/DirectRunner.java   | 14 ++-
 .../direct/ExecutorServiceParallelExecutor.java | 40 ++--
 .../beam/runners/direct/PipelineExecutor.java   |  2 +-
 .../beam/runners/direct/TransformExecutor.java  | 10 ++---
 .../runners/direct/TransformExecutorTest.java   | 16 
 6 files changed, 39 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e51c840/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
index 8e51d6f..2986df1 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CompletionCallback.java
@@ -38,7 +38,7 @@ interface CompletionCallback {
   void handleEmpty(AppliedPTransform transform);
 
   /**
-   * Handle a result that terminated abnormally due to the provided {@link 
Throwable}.
+   * Handle a result that terminated abnormally due to the provided {@link 
Exception}.
*/
-  void handleThrowable(CommittedBundle inputBundle, Throwable t);
+  void handleException(CommittedBundle inputBundle, Exception t);
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e51c840/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 6ef2472..664a915 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -403,18 +403,10 @@ public class DirectRunner
  *
  * See also {@link PipelineExecutor#awaitCompletion()}.
  */
-public State awaitCompletion() throws Throwable {
+public State awaitCompletion() throws Exception {
   if (!state.isTerminal()) {
-try {
-  executor.awaitCompletion();
-  state = State.DONE;
-} catch (InterruptedException e) {
-  Thread.currentThread().interrupt();
-  throw e;
-} catch (Throwable t) {
-  state = State.FAILED;
-  throw t;
-}
+executor.awaitCompletion();
+state = State.DONE;
   }
   return state;
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5e51c840/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 3274524..e32f671 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -234,7 +234,7 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
   }
 
   @Override
-  public void awaitCompletion() throws Throwable {
+  public void awaitCompletion() throws Exception {
 VisibleExecutorUpdate update;
 do {
   // Get an update; don't block forever if another thread has handled it
@@ -243,8 +243,8 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
 // there are no updates to process and no updates will ever be 
published because the
 // executor is shutdown
 return;
-  } else if (update != null && update.throwable.isPresent()) {
-throw update.throwable.get();
+  } else if (update != null && update.exception.isPresent()) {
+   

Jenkins build became unstable: beam_PostCommit_MavenVerify #1540

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1540

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-755) beam-runners-core-java RunnableOnService tests not executing

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-755:
--

 Summary: beam-runners-core-java RunnableOnService tests not 
executing
 Key: BEAM-755
 URL: https://issues.apache.org/jira/browse/BEAM-755
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Luke Cwik
Assignee: Frances Perry


org.apache.beam:beam-runners-core-java is not specified as an integration test 
dependency to scan within runners/pom.xml

There is also in runners/direct-java/pom.xml where its 
org.apache.beam:beam-runners-java-core and should be 
org.apache.beam:beam-runners-core-java

Finally, even if these dependencies are added and the typo fixed. When running 
the runnable on service integration tests, SplittableParDoTest which contains 
@RunnableOnService tests (part of runners/core-java) doesn't execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-722) Add Display Data to the Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576904#comment-15576904
 ] 

ASF GitHub Bot commented on BEAM-722:
-

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1112

[BEAM-722] Display data infrastructure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam 
display-data-infrastructure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1112.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1112


commit f3c7ebd24ecfd0b46aa4b2d6c906c4c1331fd13a
Author: Pablo 
Date:   2016-10-14T18:44:47Z

Laying down infrastructure for static display data




> Add Display Data to the Python SDK
> --
>
> Key: BEAM-722
> URL: https://issues.apache.org/jira/browse/BEAM-722
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Frances Perry
>
> The DisplayData feature has been added to the Java SDK (see blog post 
> announcing it: 
> https://cloud.google.com/blog/big-data/2016/06/dataflow-updates-see-more-details-about-your-pipelines).
>  We need now to add it to the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1112: [BEAM-722] Display data infrastructure

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1112

[BEAM-722] Display data infrastructure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam 
display-data-infrastructure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1112.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1112


commit f3c7ebd24ecfd0b46aa4b2d6c906c4c1331fd13a
Author: Pablo 
Date:   2016-10-14T18:44:47Z

Laying down infrastructure for static display data




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1007: Add support for bz2 compression in python

[tim1357]

Github user tim1357 closed the pull request at:

https://github.com/apache/incubator-beam/pull/1007


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1539

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1539

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-747) Text checksum verifier is not resilient to eventually consistent filesystems

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576630#comment-15576630
 ] 

Mark Liu commented on BEAM-747:
---

Yes, it's worth having retry in file path matching and reading in order to 
handle IO failures from filesystem and some special cases like no file is 
found.  

As for example2, one place to add sharding name template is the ouputpath 
argument passing to FileChecksumMatcher. Instead of using ".../result*, we can 
use ".../result*-of-*". This can avoid reading irrelevant files but can't 
guaranty all shards are read unless given total number of shards. 

The current thought in my mind is passing the number of shards from command 
line as an optional test option, then pass it to the verifier. Not sure if we 
have a better way to do that. Since from previous test results, I found that 
the number of shards is runner dependent.

> Text checksum verifier is not resilient to eventually consistent filesystems
> 
>
> Key: BEAM-747
> URL: https://issues.apache.org/jira/browse/BEAM-747
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> Example 1: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console
> Here it looks like we need to retry listing files, at least a little bit, if 
> none are found. They did show up:
> {code}
> gsutil ls 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
> {code}
> Example 2: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1525/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> Here it looks like we need to fill in the shard template if the filesystem 
> does not give us a consistent result:
> {code}
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [0 of 1] Read 162 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-0-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [1 of 1] Read 144 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-2-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> matchesSafely
> INFO: Generated checksum for output data: 
> aec68948b2515e6ea35fd1ed7649c267a10a01e5
> {code}
> We missed shard 1-of-3 and hence got the wrong checksum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1538

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1538

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1106: ignore JVM crash logs in Apache RAT

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1106


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: ignore JVM crash logs in Apache RAT

[dhalperi]

ignore JVM crash logs in Apache RAT


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/653b1309
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/653b1309
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/653b1309

Branch: refs/heads/master
Commit: 653b13095d89497e11b0c24abfc4c66446995b45
Parents: 0a413e7
Author: Dan Halperin 
Authored: Fri Oct 14 11:07:40 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 14:27:33 2016 -0700

--
 .gitignore | 3 +++
 pom.xml| 1 +
 2 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/653b1309/.gitignore
--
diff --git a/.gitignore b/.gitignore
index d6cffec..0340d8a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,6 +25,9 @@ bin/
 # committed.
 dependency-reduced-pom.xml
 
+# Hotspot VM leaves this log in a non-target directory when java crashes
+hs_err_pid*.log
+
 # Ignore files that end with '~', since they are most likely auto-save files
 # produced by a text editor.
 *~

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/653b1309/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8a28bd9..1263827 100644
--- a/pom.xml
+++ b/pom.xml
@@ -874,6 +874,7 @@
   
   **/target/**/*
   **/dependency-reduced-pom.xml
+  **/hs_err_pid*.log
   .github/**/*
   **/*.iml
   **/package-list

[1/2] incubator-beam git commit: Perform initial splitting in the DirectRunner

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 86d222aab -> 0a413e78e


Perform initial splitting in the DirectRunner

This allows sources to be read from in parallel and generates initial
splits.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f68fea02
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f68fea02
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f68fea02

Branch: refs/heads/master
Commit: f68fea02b63e5844b9ccbd31ff8e02da407f65b7
Parents: 86d222a
Author: Thomas Groh 
Authored: Wed Oct 5 16:11:21 2016 -0700
Committer: Luke Cwik 
Committed: Fri Oct 14 13:54:55 2016 -0700

--
 .../direct/BoundedReadEvaluatorFactory.java |  40 ++-
 .../beam/runners/direct/DirectOptions.java  |  23 ++
 .../beam/runners/direct/DirectRunner.java   |  11 +-
 .../beam/runners/direct/EmptyInputProvider.java |  12 +-
 .../direct/ExecutorServiceParallelExecutor.java |  15 +-
 .../beam/runners/direct/RootInputProvider.java  |   7 +-
 .../runners/direct/RootProviderRegistry.java|   5 +-
 .../direct/TestStreamEvaluatorFactory.java  |   4 +-
 .../direct/TransformEvaluatorRegistry.java  |  10 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |  35 ++-
 .../beam/runners/direct/WatermarkManager.java   |   1 +
 .../direct/BoundedReadEvaluatorFactoryTest.java |  41 ++-
 .../direct/FlattenEvaluatorFactoryTest.java |   9 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |   2 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |  55 +++-
 .../sdk/io/gcp/bigquery/BigQueryAvroUtils.java  |  69 -
 .../io/gcp/bigquery/BigQueryAvroUtilsTest.java  | 132 +++--
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 292 ++-
 .../sdk/io/gcp/bigtable/BigtableIOTest.java |   9 +-
 19 files changed, 662 insertions(+), 110 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f68fea02/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 326a535..843dcd6 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -18,28 +18,32 @@
 package org.apache.beam.runners.direct;
 
 import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
 import java.io.IOException;
 import java.util.Collection;
-import java.util.Collections;
+import java.util.List;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
 import org.apache.beam.runners.direct.StepTransformResult.Builder;
 import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
-import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.Read.Bounded;
+import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.PCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * A {@link TransformEvaluatorFactory} that produces {@link TransformEvaluator 
TransformEvaluators}
  * for the {@link Bounded Read.Bounded} primitive {@link PTransform}.
  */
 final class BoundedReadEvaluatorFactory implements TransformEvaluatorFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BoundedReadEvaluatorFactory.class);
   private final EvaluationContext evaluationContext;
 
   BoundedReadEvaluatorFactory(EvaluationContext evaluationContext) {
@@ -126,18 +130,32 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 @Override
-public Collection 
getInitialInputs(AppliedPTransform transform) {
-  return createInitialSplits((AppliedPTransform) transform);
+public Collection getInitialInputs(
+AppliedPTransform transform, int targetParallelism) throws 
Exception {
+  return createInitialSplits((AppliedPTransform) transform, 
targetParallelism);
 }
 
-private  Collection createInitialSplits(
-AppliedPTransform> transform) {
+private 
+

[GitHub] incubator-beam pull request #1063: [BEAM-310] Perform initial splitting in t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1063


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-310] Perform initial splitting in the DirectRunner

[lcwik]

[BEAM-310] Perform initial splitting in the DirectRunner

This closes #1063


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a413e78
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a413e78
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a413e78

Branch: refs/heads/master
Commit: 0a413e78e8f937ab3c221b78be25037021685e0d
Parents: 86d222a f68fea0
Author: Luke Cwik 
Authored: Fri Oct 14 13:55:24 2016 -0700
Committer: Luke Cwik 
Committed: Fri Oct 14 13:55:24 2016 -0700

--
 .../direct/BoundedReadEvaluatorFactory.java |  40 ++-
 .../beam/runners/direct/DirectOptions.java  |  23 ++
 .../beam/runners/direct/DirectRunner.java   |  11 +-
 .../beam/runners/direct/EmptyInputProvider.java |  12 +-
 .../direct/ExecutorServiceParallelExecutor.java |  15 +-
 .../beam/runners/direct/RootInputProvider.java  |   7 +-
 .../runners/direct/RootProviderRegistry.java|   5 +-
 .../direct/TestStreamEvaluatorFactory.java  |   4 +-
 .../direct/TransformEvaluatorRegistry.java  |  10 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |  35 ++-
 .../beam/runners/direct/WatermarkManager.java   |   1 +
 .../direct/BoundedReadEvaluatorFactoryTest.java |  41 ++-
 .../direct/FlattenEvaluatorFactoryTest.java |   9 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |   2 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |  55 +++-
 .../sdk/io/gcp/bigquery/BigQueryAvroUtils.java  |  69 -
 .../io/gcp/bigquery/BigQueryAvroUtilsTest.java  | 132 +++--
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 292 ++-
 .../sdk/io/gcp/bigtable/BigtableIOTest.java |   9 +-
 19 files changed, 662 insertions(+), 110 deletions(-)
--

[jira] [Commented] (BEAM-310) Exercise splitIntoBundles/generateInitialSplits in the Direct Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576473#comment-15576473
 ] 

ASF GitHub Bot commented on BEAM-310:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1063


> Exercise splitIntoBundles/generateInitialSplits in the Direct Runner
> 
>
> Key: BEAM-310
> URL: https://issues.apache.org/jira/browse/BEAM-310
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> BoundedSource#splitIntoBundles and UnboundedSource#generateInitialSplits are 
> the methods by which sources can be accessed in parallel. Exercising these 
> methods allows reads (and all transforms downstream) to be executed in 
> parallel both pre and post a GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[Jeffrey Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576406#comment-15576406
 ] 

Jeffrey Payne commented on BEAM-697:


Submitted fix in PR https://github.com/apache/incubator-beam/pull/

> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576401#comment-15576401
 ] 

ASF GitHub Bot commented on BEAM-697:
-

GitHub user jeffkpayne opened a pull request:

https://github.com/apache/incubator-beam/pull/

[BEAM-697] TextIO.Write.Bound.withHeader() and withFooter() now use t…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…he current value of validate for the returned Bound instance, added 
supporting tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bomboradata/incubator-beam BEAM-697-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #


commit 5cc0441d1e2e57be6a2b7c80d4c477657f004104
Author: Jeffrey Scott Keone Payne 
Date:   2016-10-14T20:33:16Z

[BEAM-697] TextIO.Write.Bound.withHeader() and withFooter() now use the 
current value of validate for the returned Bound instance, added supporting 
tests




> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1111: [BEAM-697] TextIO.Write.Bound.withHeader(...

[jeffkpayne]

GitHub user jeffkpayne opened a pull request:

https://github.com/apache/incubator-beam/pull/

[BEAM-697] TextIO.Write.Bound.withHeader() and withFooter() now use t…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…he current value of validate for the returned Bound instance, added 
supporting tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bomboradata/incubator-beam BEAM-697-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #


commit 5cc0441d1e2e57be6a2b7c80d4c477657f004104
Author: Jeffrey Scott Keone Payne 
Date:   2016-10-14T20:33:16Z

[BEAM-697] TextIO.Write.Bound.withHeader() and withFooter() now use the 
current value of validate for the returned Bound instance, added supporting 
tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-722) Add Display Data to the Python SDK

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576336#comment-15576336
 ] 

Pablo Estrada commented on BEAM-722:


I am working on adding this feature. I'm basically mirroring the way in which 
we do this in Java, but trying to make it a bit more Pythonic. Here's a quick 
example of how this would work:

{code:title=display_data_example.py|borderStyle=solid}
class MyParDo(beam.PTransform):
  def display_data(self):
return {'disp_data_key': MyParDo,
'loneliest_number': 1,
'secret_url': {'value': 'awebsite.com', 'url': 
'http://awebsite.com'} ,
'fn': {'value': self.fn, 'label': 'Display data of underlying DoFn'}
   }

{code}

I'm renaming the populateDisplayData function to display_data, and instead of 
using a builder, I return a dictionary of key:value pairs - and if a user wants 
to specify more parameters than just the key:value, they can pass a dictionary 
with them. Also, if the user passes as value an object that inherits from the 
HasDisplayData class, then that object's display data will be included.

I have a [small 
commit|https://github.com/pabloem/incubator-beam/commit/f3c7ebd24ecfd0b46aa4b2d6c906c4c1331fd13a]
 that adds this. You can see some [examples in the few 
unittests|https://github.com/pabloem/incubator-beam/commit/f3c7ebd24ecfd0b46aa4b2d6c906c4c1331fd13a#diff-74a8ae565b6cf2631423124a587c2beaR1].

If everyone is okay with this, I'll add comments, tests, and address feedback.

> Add Display Data to the Python SDK
> --
>
> Key: BEAM-722
> URL: https://issues.apache.org/jira/browse/BEAM-722
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Frances Perry
>
> The DisplayData feature has been added to the Java SDK (see blog post 
> announcing it: 
> https://cloud.google.com/blog/big-data/2016/06/dataflow-updates-see-more-details-about-your-pipelines).
>  We need now to add it to the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: Closes #1017

[robertwb]

Closes #1017


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/56ab1a4d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/56ab1a4d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/56ab1a4d

Branch: refs/heads/python-sdk
Commit: 56ab1a4d59e66eea5482caf53850b134e55e53e8
Parents: 68ddb7e 36ea9b4
Author: Robert Bradshaw 
Authored: Fri Oct 14 12:43:42 2016 -0700
Committer: Robert Bradshaw 
Committed: Fri Oct 14 12:43:42 2016 -0700

--
 sdks/python/README.md |  2 +-
 sdks/python/setup.py  | 30 ++
 2 files changed, 31 insertions(+), 1 deletion(-)
--

[1/3] incubator-beam git commit: Fixed pip requirement.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 68ddb7ecf -> 56ab1a4d5


Fixed pip requirement.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b5061a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b5061a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b5061a1

Branch: refs/heads/python-sdk
Commit: 2b5061a139169f86531727f3d20c8ee3581626f1
Parents: 68ddb7e
Author: mbuccini 
Authored: Tue Sep 27 10:30:38 2016 +0200
Committer: Robert Bradshaw 
Committed: Fri Oct 14 12:40:54 2016 -0700

--
 sdks/python/README.md | 2 +-
 sdks/python/setup.py  | 7 ++-
 2 files changed, 7 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b5061a1/sdks/python/README.md
--
diff --git a/sdks/python/README.md b/sdks/python/README.md
index dca9517..e003aab 100644
--- a/sdks/python/README.md
+++ b/sdks/python/README.md
@@ -118,7 +118,7 @@ set up your machine's Python development environment.
  Install ``pip``
 
 `pip` is Python's package manager.  If you already have `pip` installed
-(type `pip -V` to check), skip this step.
+(type `pip -V` to check), please make sure to have at least version 7.0.0.
 
 There are several ways to install `pip`; use whichever works for you.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b5061a1/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 09ceef9..20870b8 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -17,9 +17,12 @@
 
 """Apache Beam SDK for Python setup file."""
 
+from distutils.version import StrictVersion
+
 import os
 import platform
 import setuptools
+import pkg_resources
 
 
 def get_version():
@@ -40,6 +43,9 @@ PACKAGE_LONG_DESCRIPTION = '''
 TBD
 '''
 
+_PIP_VERSION = pkg_resources.get_distribution('pip').version
+assert StrictVersion(_PIP_VERSION) >= StrictVersion('0.7.0'), \
+"This SDK requires 'pip' >= 7.0.0"
 
 # Currently all compiled modules are optional  (for performance only).
 if platform.system() == 'Windows':
@@ -65,7 +71,6 @@ REQUIRED_PACKAGES = [
 'pyyaml>=3.10',
 ]
 
-
 setuptools.setup(
 name=PACKAGE_NAME,
 version=PACKAGE_VERSION,

[2/3] incubator-beam git commit: Added cython version check

[robertwb]

Added cython version check

Replaced assertions with warnings.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/36ea9b4f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/36ea9b4f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/36ea9b4f

Branch: refs/heads/python-sdk
Commit: 36ea9b4f974a106ac19953ee01d2cdcf8ad53b40
Parents: 2b5061a
Author: markon 
Authored: Thu Sep 29 20:45:18 2016 +0200
Committer: Robert Bradshaw 
Committed: Fri Oct 14 12:41:13 2016 -0700

--
 sdks/python/setup.py | 33 +
 1 file changed, 29 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/36ea9b4f/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 20870b8..b2d22ff 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -21,8 +21,11 @@ from distutils.version import StrictVersion
 
 import os
 import platform
+import warnings
+
 import setuptools
-import pkg_resources
+
+from pkg_resources import get_distribution, DistributionNotFound
 
 
 def get_version():
@@ -43,9 +46,30 @@ PACKAGE_LONG_DESCRIPTION = '''
 TBD
 '''
 
-_PIP_VERSION = pkg_resources.get_distribution('pip').version
-assert StrictVersion(_PIP_VERSION) >= StrictVersion('0.7.0'), \
-"This SDK requires 'pip' >= 7.0.0"
+REQUIRED_PIP_VERSION = '7.0.0'
+_PIP_VERSION = get_distribution('pip').version
+if StrictVersion(_PIP_VERSION) < StrictVersion(REQUIRED_PIP_VERSION):
+  warnings.warn(
+  "You are using version {0} of pip. " \
+  "However, version {1} is recommended.".format(
+  _PIP_VERSION, REQUIRED_PIP_VERSION
+  )
+  )
+
+
+REQUIRED_CYTHON_VERSION = '0.23.2'
+try:
+  _CYTHON_VERSION = get_distribution('cython').version
+  if StrictVersion(_CYTHON_VERSION) < StrictVersion(REQUIRED_CYTHON_VERSION):
+warnings.warn(
+"You are using version {0} of cython. " \
+"However, version {1} is recommended.".format(
+_CYTHON_VERSION, REQUIRED_CYTHON_VERSION
+)
+)
+except DistributionNotFound:
+  # do nothing if Cython is not installed
+  pass
 
 # Currently all compiled modules are optional  (for performance only).
 if platform.system() == 'Windows':
@@ -71,6 +95,7 @@ REQUIRED_PACKAGES = [
 'pyyaml>=3.10',
 ]
 
+
 setuptools.setup(
 name=PACKAGE_NAME,
 version=PACKAGE_VERSION,

[jira] [Issue Comment Deleted] (BEAM-520) Update Python SDK example tests to use assert_that

[Geetha Bijjam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geetha Bijjam updated BEAM-520:
---
Comment: was deleted

(was: Travis Build Failed for BEAM-520)

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-520) Update Python SDK example tests to use assert_that

[Geetha Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576278#comment-15576278
 ] 

Geetha Bijjam commented on BEAM-520:


Travis Build Failed for BEAM-520

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-753) Travis failure (cannot import name locked_file)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576262#comment-15576262
 ] 

ASF GitHub Bot commented on BEAM-753:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1110


> Travis failure (cannot import name locked_file)
> ---
>
> Key: BEAM-753
> URL: https://issues.apache.org/jira/browse/BEAM-753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> ERROR: Failure: ImportError (cannot import name locked_file)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
>  line 31, in 
> from apache_beam.io import concat_source
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
>  line 24, in 
> from apache_beam.io import iobase
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
>  line 818, in 
> from apache_beam.runners.dataflow.native_io.iobase import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
>  line 23, in 
> from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 43, in 
> from apache_beam.internal.clients import dataflow as dataflow_api
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
>  line 23, in 
> from apitools.base.py import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 22, in 
> from apitools.base.py.credentials_lib import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
>  line 50, in 
> from oauth2client import locked_file
> ImportError: cannot import name locked_file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1110: [BEAM-753] change required version for oa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1110


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: change required version for oauth2client

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk eb57b974c -> 68ddb7ecf


change required version for oauth2client


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3727fb81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3727fb81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3727fb81

Branch: refs/heads/python-sdk
Commit: 3727fb816a8cf8f802840e77a49151b62c10738e
Parents: eb57b97
Author: Ahmet Altay 
Authored: Fri Oct 14 12:07:49 2016 -0700
Committer: Ahmet Altay 
Committed: Fri Oct 14 12:07:49 2016 -0700

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3727fb81/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 58e677e..09ceef9 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -59,7 +59,7 @@ REQUIRED_PACKAGES = [
 'google-apitools>=0.5.2',
 'httplib2>=0.8',
 'mock>=1.0.1',
-'oauth2client>=2.0.1',
+'oauth2client>=2.0.1,<4.0.0',
 'protorpc>=0.9.1',
 'python-gflags>=2.0',
 'pyyaml>=3.10',

[2/2] incubator-beam git commit: Closes #1110

[robertwb]

Closes #1110


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/68ddb7ec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/68ddb7ec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/68ddb7ec

Branch: refs/heads/python-sdk
Commit: 68ddb7ecf99b8c1b1c745d7a09abbb4b69fa15b3
Parents: eb57b97 3727fb8
Author: Robert Bradshaw 
Authored: Fri Oct 14 12:27:15 2016 -0700
Committer: Robert Bradshaw 
Committed: Fri Oct 14 12:27:15 2016 -0700

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1537

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-753) Travis failure (cannot import name locked_file)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576219#comment-15576219
 ] 

ASF GitHub Bot commented on BEAM-753:
-

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1110

[BEAM-753] change required version for oauth2client

There was a new oauth2client release to PyPI this morning, it brake sdk 
installation and tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1110


commit 3727fb816a8cf8f802840e77a49151b62c10738e
Author: Ahmet Altay 
Date:   2016-10-14T19:07:49Z

change required version for oauth2client




> Travis failure (cannot import name locked_file)
> ---
>
> Key: BEAM-753
> URL: https://issues.apache.org/jira/browse/BEAM-753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> ERROR: Failure: ImportError (cannot import name locked_file)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
>  line 31, in 
> from apache_beam.io import concat_source
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
>  line 24, in 
> from apache_beam.io import iobase
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
>  line 818, in 
> from apache_beam.runners.dataflow.native_io.iobase import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
>  line 23, in 
> from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 43, in 
> from apache_beam.internal.clients import dataflow as dataflow_api
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
>  line 23, in 
> from apitools.base.py import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 22, in 
> from apitools.base.py.credentials_lib import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
>  line 50, in 
> from oauth2client import locked_file
> ImportError: cannot import name locked_file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1110: [BEAM-753] change required version for oa...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1110

[BEAM-753] change required version for oauth2client

There was a new oauth2client release to PyPI this morning, it brake sdk 
installation and tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1110


commit 3727fb816a8cf8f802840e77a49151b62c10738e
Author: Ahmet Altay 
Date:   2016-10-14T19:07:49Z

change required version for oauth2client




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-753) Travis failure (cannot import name locked_file)

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576178#comment-15576178
 ] 

Ahmet Altay commented on BEAM-753:
--

There was a new oauth2client release to PyPI this morning, it brake sdk 
installation and tests. 

Python SDK setup.py list this requirement:
oauth2client>=2.0.1

There should be two updates:
1. Short term, change setup.py to fix the break and work with an older version 
of oauth2client, the previous version 3.0.0 was working fine.
2. Understand the difference and update the code to work with oauth2client 4.0.0

> Travis failure (cannot import name locked_file)
> ---
>
> Key: BEAM-753
> URL: https://issues.apache.org/jira/browse/BEAM-753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> ERROR: Failure: ImportError (cannot import name locked_file)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
>  line 31, in 
> from apache_beam.io import concat_source
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
>  line 24, in 
> from apache_beam.io import iobase
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
>  line 818, in 
> from apache_beam.runners.dataflow.native_io.iobase import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
>  line 23, in 
> from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 43, in 
> from apache_beam.internal.clients import dataflow as dataflow_api
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
>  line 23, in 
> from apitools.base.py import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 22, in 
> from apitools.base.py.credentials_lib import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
>  line 50, in 
> from oauth2client import locked_file
> ImportError: cannot import name locked_file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1109: "Intern" schemas and schema strings to pr...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1109

"Intern" schemas and schema strings to prevent out of memory issues

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

"Intern" schemas and schema strings to prevent out of memory issues when 
dealing with many many files in Avro sources.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam intern_avro_schema

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1109.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1109


commit 867b20751449a2a2686822e3418cbd4efc65cc7c
Author: Luke Cwik 
Date:   2016-10-14T18:46:37Z

"Intern" schemas and schema strings to prevent out of memory issues when 
dealing with many many files in Avro sources.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-754) WordCountIT Flake -- Incorrect checksum

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-754:
-

 Summary: WordCountIT Flake -- Incorrect checksum
 Key: BEAM-754
 URL: https://issues.apache.org/jira/browse/BEAM-754
 Project: Beam
  Issue Type: Bug
Reporter: Jason Kuster
Assignee: Mark Liu
Priority: Minor


WordCountIT flaked in Jenkins PostCommit -- 
https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1532/testReport/junit/org.apache.beam.examples/WordCountIT/
 -- due to an incorrect checksum. Can we add some additional debug output so we 
can understand what happened in these cases?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-520) Update Python SDK example tests to use assert_that

[Geetha Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576063#comment-15576063
 ] 

Geetha Bijjam edited comment on BEAM-520 at 10/14/16 6:41 PM:
--

Thanks for the info.
I am working on multiple_output_pardo_test.py.

PR: (https://github.com/apache/incubator-beam/pull/1108)

Travis build in failing in a totally different file: /bin/python 
apache_beam/examples/complete/autocomplete_test.py

Please find the travis logs here:
https://api.travis-ci.org/jobs/167721614/log.txt?deansi=true


was (Author: venbijjam):
Thanks for the info.
I am working on multiple_output_pardo_test.py and will send a PR once the 
Travis Build succeeds.

Travis build in failing in a totally different file: /bin/python 
apache_beam/examples/complete/autocomplete_test.py

Please find the travis logs here:
https://api.travis-ci.org/jobs/167721614/log.txt?deansi=true

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-520) Update Python SDK example tests to use assert_that

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576148#comment-15576148
 ] 

Ahmet Altay commented on BEAM-520:
--

Thank you for noticing this. I can reproduce the error at head. This is not 
related to your change. I created 
(https://issues.apache.org/jira/browse/BEAM-753) for this and looking at it.

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-753) Travis failure (cannot import name locked_file)

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-753:


 Summary: Travis failure (cannot import name locked_file)
 Key: BEAM-753
 URL: https://issues.apache.org/jira/browse/BEAM-753
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay


ERROR: Failure: ImportError (cannot import name locked_file)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
 line 418, in loadTestsFromName
addr.filename, addr.module)
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
 line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
 line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
 line 78, in 
from apache_beam import io
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
 line 21, in 
from apache_beam.io.avroio import *
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
 line 29, in 
from apache_beam.io import filebasedsource
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
 line 31, in 
from apache_beam.io import concat_source
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
 line 24, in 
from apache_beam.io import iobase
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
 line 818, in 
from apache_beam.runners.dataflow.native_io.iobase import *
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
 line 23, in 
from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
 line 43, in 
from apache_beam.internal.clients import dataflow as dataflow_api
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
 line 23, in 
from apitools.base.py import *
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
 line 22, in 
from apitools.base.py.credentials_lib import *
  File 
"/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
 line 50, in 
from oauth2client import locked_file
ImportError: cannot import name locked_file




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1108: Beam-520 Update Python SDK example tests ...

[venbijjam]

GitHub user venbijjam opened a pull request:

https://github.com/apache/incubator-beam/pull/1108

Beam-520 Update Python SDK example tests to use assert_that (file: 
multiple_output_pardo_test.py)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/venbijjam/incubator-beam BEAM-520

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1108


commit 4e7f870d7bdeba3fda27f4e43c0aec8341d282ba
Author: venbijjam 
Date:   2016-10-14T18:27:38Z

BEAM-520

commit 5ca79cd40558ea46b6725862f4fe3b6cbef01c44
Author: venbijjam 
Date:   2016-10-14T18:29:48Z

Update multiple_output_pardo_test.py

commit de6619b2201824cf8cc5c98d9f5fee2a2d41d9c9
Author: venbijjam 
Date:   2016-10-14T18:30:56Z

Update multiple_output_pardo_test.py




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify #1536

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1536

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-520) Update Python SDK example tests to use assert_that

[Geetha Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576063#comment-15576063
 ] 

Geetha Bijjam edited comment on BEAM-520 at 10/14/16 6:34 PM:
--

Thanks for the info.
I am working on multiple_output_pardo_test.py and will send a PR once the 
Travis Build succeeds.

Travis build in failing in a totally different file: /bin/python 
apache_beam/examples/complete/autocomplete_test.py

Please find the travis logs here:
https://api.travis-ci.org/jobs/167721614/log.txt?deansi=true


was (Author: venbijjam):
Thanks for the info.
I am working on multiple_output_pardo_test.py and will send a PR once the 
Travis Build succeeds.

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1335

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_MavenVerify #1535

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-751) infrastructure for extracting code snippets into documentation

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-751:
---
Issue Type: Improvement  (was: Bug)

> infrastructure for extracting code snippets into documentation
> --
>
> Key: BEAM-751
> URL: https://issues.apache.org/jira/browse/BEAM-751
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: James Malone
>  Labels: starter
>
> As we fill in more and more documentation, the number of code snippets is 
> going to drastically increase, and we should ensure the quality of those 
> snippets by automatically extracting them from code that is regularly 
> compiled and tested.  
> Goals:
> * automatically extract code snippets from incubator-beam for use in the beam 
> website documentation
> * use stable references so folks editing the code can clearly tell what 
> documentation changes this will result in (good: specially formatted comment, 
> bad: line number)
> * freshness (is live possible? or at least during the general 'jekyll build' 
> phase?)
> The best we've found so far is using jekyll-gist with gist-it, but that would 
> rely on fragile line numbers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-749) Syntax highlight on website

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-749:
---
Issue Type: Improvement  (was: Bug)

> Syntax highlight on website
> ---
>
> Key: BEAM-749
> URL: https://issues.apache.org/jira/browse/BEAM-749
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: James Malone
>
> We should able to enable rouge on the website in order to get syntax 
> highlighting in the programming guide, walkthroughs, etc.
> https://jekyllrb.com/docs/templates/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-752) infrastructure for toggling code snippets in documentation

[Frances Perry (JIRA)]

Frances Perry created BEAM-752:
--

 Summary: infrastructure for toggling code snippets in documentation
 Key: BEAM-752
 URL: https://issues.apache.org/jira/browse/BEAM-752
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Frances Perry
Assignee: James Malone


Once the python sdk gets merged to the master branch, a lot of our 
documentation (programming guide, walkthroughs, etc) will need to support 
multiple languages.

The hope is that the vast bulk of the prose can be written about Beam concepts 
in a language independent way. But for code snippets it would be great to be 
able to toggle languages.

Goals:
* Support tabbed language toggles for both code and small sections of text.
* Support easily changing the default per-user-visit so that the entire file 
(or even better entire site) defaults to showing a specific language



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-751) infrastructure for extracting code snippets into documentation

[Frances Perry (JIRA)]

Frances Perry created BEAM-751:
--

 Summary: infrastructure for extracting code snippets into 
documentation
 Key: BEAM-751
 URL: https://issues.apache.org/jira/browse/BEAM-751
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


As we fill in more and more documentation, the number of code snippets is going 
to drastically increase, and we should ensure the quality of those snippets by 
automatically extracting them from code that is regularly compiled and tested.  

Goals:
* automatically extract code snippets from incubator-beam for use in the beam 
website documentation
* use stable references so folks editing the code can clearly tell what 
documentation changes this will result in (good: specially formatted comment, 
bad: line number)
* freshness (is live possible? or at least during the general 'jekyll build' 
phase?)

The best we've found so far is using jekyll-gist with gist-it, but that would 
rely on fragile line numbers.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1107: [BEAM-551] Add property name to RuntimeVa...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1107

[BEAM-551] Add property name to RuntimeValueProvider

The intention here is to allow Native IO to perform the same substitution 
that would otherwise take place at job execution time.

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam prop_name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1107.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1107


commit e84cb0911129757e42716cce5c75b1d64e619e31
Author: sammcveety 
Date:   2016-10-14T17:41:33Z

Add property name to RuntimeValueProvider




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1106: ignore JVM crash logs in Apache RAT

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1106

ignore JVM crash logs in Apache RAT

R: @jasonkuster



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam rat-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1106.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1106


commit 8e2e99ffa126cd1b6cffc42481b4277cdfba5fa4
Author: Dan Halperin 
Date:   2016-10-14T18:07:40Z

ignore JVM crash logs in Apache RAT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1532

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1532

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1532

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1052: Delete DatastoreWordCount

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1052


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1052

[dhalperi]

Closes #1052


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/86d222aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/86d222aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/86d222aa

Branch: refs/heads/master
Commit: 86d222aab35d30eb314fb0ef8e547ed4aaf8f803
Parents: 87c5839 d8ca34c
Author: Dan Halperin 
Authored: Fri Oct 14 10:44:32 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 10:44:32 2016 -0700

--
 examples/java/pom.xml   |   5 -
 .../examples/cookbook/DatastoreWordCount.java   | 261 ---
 2 files changed, 266 deletions(-)
--

[1/2] incubator-beam git commit: Delete DatastoreWordCount

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 87c5839dc -> 86d222aab


Delete DatastoreWordCount

This is the kind of example we do not need to have in Beam. It's just
WordCount with a different data source.

Also remove a no-longer-needed declared dependency.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d8ca34cc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d8ca34cc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d8ca34cc

Branch: refs/heads/master
Commit: d8ca34ccad3ba0717b3a0381936fc9bc11b3bbad
Parents: 87c5839
Author: Daniel Halperin 
Authored: Tue Oct 4 19:23:54 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 10:44:31 2016 -0700

--
 examples/java/pom.xml   |   5 -
 .../examples/cookbook/DatastoreWordCount.java   | 261 ---
 2 files changed, 266 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d8ca34cc/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index d18f959..37cb15a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -466,11 +466,6 @@
 
 
 
-  com.google.code.findbugs
-  annotations
-
-
-
   com.google.cloud.datastore
   datastore-v1-proto-client
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d8ca34cc/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
deleted file mode 100644
index c0066e6..000
--- 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
+++ /dev/null
@@ -1,261 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.examples.cookbook;
-
-import static com.google.datastore.v1.client.DatastoreHelper.getString;
-import static com.google.datastore.v1.client.DatastoreHelper.makeFilter;
-import static com.google.datastore.v1.client.DatastoreHelper.makeKey;
-import static com.google.datastore.v1.client.DatastoreHelper.makeValue;
-
-import com.google.datastore.v1.Entity;
-import com.google.datastore.v1.Key;
-import com.google.datastore.v1.PropertyFilter;
-import com.google.datastore.v1.Query;
-import com.google.datastore.v1.Value;
-import java.util.Map;
-import java.util.UUID;
-import javax.annotation.Nullable;
-import org.apache.beam.examples.WordCount;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.io.gcp.datastore.DatastoreIO;
-import org.apache.beam.sdk.io.gcp.datastore.DatastoreV1;
-import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.options.Validation;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-
-/**
- * A WordCount example using DatastoreIO.
- *
- * This example shows how to use DatastoreIO to read from Cloud Datastore 
and
- * write the results to Cloud Storage.  Note that this example will write
- * data to Cloud Datastore, which may incur charge for Cloud Datastore 
operations.
- *
- * To run this example, users need to use gcloud to get credential for 
Cloud Datastore:
- * {@code
- * $ gcloud auth login
- * }
- *
- * To run this pipeline locally, the following options must be provided:
- * {@code
- *   --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PATH]
- * }
- *
- * To change the runner, specify:
- * {@code
- *   --runner=YOUR_SELECTED_RUNNER
- * }
- * 
- 

Build failed in Jenkins: beam_PostCommit_MavenVerify #1534

[Apache Jenkins Server]

See 

Changes:

[klk] Remove getOrCreate pattern from DoFnSignatures and DoFnInvokers

--
[EnvInject] - Mask passwords passed as build parameters.
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 87c5839dc859bfd9f404cee9af6eacf01ffb7686 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 87c5839dc859bfd9f404cee9af6eacf01ffb7686
 > git rev-list 2584ebeb8c57314288a5eba480a8bd2c7dc1b1d2 # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Modules changed, recalculating dependency graph
Established TCP socket on 43622
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_MavenVerify@2] $ /home/jenkins/tools/java/latest1.8/bin/java 
-Xmx2g -Xms256m -XX:MaxPermSize=512m -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 43622
ERROR: Failed to parse POMs
java.io.IOException: Cannot run program 
"/home/jenkins/tools/java/latest1.8/bin/java" (in directory 
": error=2, No 
such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at hudson.Proc$LocalProc.(Proc.java:240)
at hudson.Proc$LocalProc.(Proc.java:212)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:815)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1148)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to beam1(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:928)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at 
hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:281)
at hudson.maven.ProcessCache.get(ProcessCache.java:236)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:778)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1741)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:531)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
at hudson.Proc$LocalProc.(Proc.java:240)
at 

[jira] [Resolved] (BEAM-55) Allow users to compress FileBasedSink output files

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-55.
-
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Allow users to compress FileBasedSink output files
> --
>
> Key: BEAM-55
> URL: https://issues.apache.org/jira/browse/BEAM-55
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Jeffrey Payne
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option 
> for compressing its output.
> In general, we discourage compression because it limits or blocks scalably 
> reading from a file in parallel. However, users may want it -- so we should 
> support the option (with appropriate warnings).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1104: [BEAM-469] Remove Remaining Nested Contex...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1104

[BEAM-469] Remove Remaining Nested Contexts from NullableCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam nullable_non_nested

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1104


commit 4e4e9dfd8a24f614da657dd31b64cea2f4f1aa03
Author: Thomas Groh 
Date:   2016-10-14T17:34:55Z

Remove Remaining Nested Contexts from NullableCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-55) Allow users to compress FileBasedSink output files

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-55:

Assignee: Jeffrey Payne

> Allow users to compress FileBasedSink output files
> --
>
> Key: BEAM-55
> URL: https://issues.apache.org/jira/browse/BEAM-55
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Jeffrey Payne
>Priority: Minor
>
> FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option 
> for compressing its output.
> In general, we discourage compression because it limits or blocks scalably 
> reading from a file in parallel. However, users may want it -- so we should 
> support the option (with appropriate warnings).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1105: [BEAM-551] Add utility to handle JSON opt...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1105

[BEAM-551] Add utility to handle JSON option manipulation

Provide a utility to be used with RuntimeValueProviders that allows runners 
to manipulate PipelineOptions by interleaving runtime parameters.

Green travis: https://travis-ci.org/sammcveety/incubator-beam/jobs/167702097

R: @dhalperi

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/json_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1105


commit 83230a2c47b533162cdbf45580daf59896149487
Author: sammcveety 
Date:   2016-10-14T17:06:08Z

Add utility to handle JSON option manipulation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575968#comment-15575968
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1105

[BEAM-551] Add utility to handle JSON option manipulation

Provide a utility to be used with RuntimeValueProviders that allows runners 
to manipulate PipelineOptions by interleaving runtime parameters.

Green travis: https://travis-ci.org/sammcveety/incubator-beam/jobs/167702097

R: @dhalperi

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/json_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1105


commit 83230a2c47b533162cdbf45580daf59896149487
Author: sammcveety 
Date:   2016-10-14T17:06:08Z

Add utility to handle JSON option manipulation




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder should encode using the context given and not always use the nested context

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575967#comment-15575967
 ] 

ASF GitHub Bot commented on BEAM-469:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1104

[BEAM-469] Remove Remaining Nested Contexts from NullableCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam nullable_non_nested

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1104


commit 4e4e9dfd8a24f614da657dd31b64cea2f4f1aa03
Author: Thomas Groh 
Date:   2016-10-14T17:34:55Z

Remove Remaining Nested Contexts from NullableCoder




> NullableCoder should encode using the context given and not always use the 
> nested context
> -
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Remove getOrCreate pattern from DoFnSignatures and DoFnInvokers

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2584ebeb8 -> 87c5839dc


Remove getOrCreate pattern from DoFnSignatures and DoFnInvokers

This excessive verbosity is also an abstraction violation. It
is an implementation detail how the value is produced.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/546a572d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/546a572d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/546a572d

Branch: refs/heads/master
Commit: 546a572d21822e38a16f0b26ef15a60fb163d2da
Parents: 49f9444
Author: Kenneth Knowles 
Authored: Fri Oct 14 08:48:49 2016 -0700
Committer: Kenneth Knowles 
Committed: Fri Oct 14 08:57:39 2016 -0700

--
 .../beam/runners/core/SplittableParDo.java  |  2 +-
 .../runners/direct/ParDoOverrideFactory.java|  2 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |  2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  4 +--
 .../sdk/transforms/reflect/DoFnInvokers.java| 12 ---
 .../sdk/transforms/reflect/DoFnSignatures.java  |  6 ++--
 .../DoFnSignaturesProcessElementTest.java   | 12 +++
 .../DoFnSignaturesSplittableDoFnTest.java   | 34 ++--
 .../transforms/reflect/DoFnSignaturesTest.java  | 16 -
 9 files changed, 46 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/546a572d/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 7645149..33d0ab7 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -92,7 +92,7 @@ public class SplittableParDo<
   public SplittableParDo(DoFn fn) {
 checkNotNull(fn, "fn must not be null");
 this.fn = fn;
-this.signature = 
DoFnSignatures.INSTANCE.getOrParseSignature(fn.getClass());
+this.signature = DoFnSignatures.INSTANCE.getSignature(fn.getClass());
 checkArgument(signature.processElement().isSplittable(), "fn must be a 
splittable DoFn");
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/546a572d/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
index a57735c..0881868 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
@@ -46,7 +46,7 @@ class ParDoOverrideFactory implements 
PTransformOverrideFactory {
   // This is an OldDoFn, hence not splittable.
   return transform;
 }
-DoFnSignature signature = 
DoFnSignatures.INSTANCE.getOrParseSignature(fn.getClass());
+DoFnSignature signature = 
DoFnSignatures.INSTANCE.getSignature(fn.getClass());
 if (!signature.processElement().isSplittable()) {
   return transform;
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/546a572d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index f671a67..12d4824 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -61,7 +61,7 @@ public class DoFnAdapters {
   /** Creates an {@link OldDoFn} that delegates to the {@link DoFn}. */
   @SuppressWarnings({"unchecked", "rawtypes"})
   public static  OldDoFn 
toOldDoFn(DoFn fn) {
-DoFnSignature signature = 
DoFnSignatures.INSTANCE.getOrParseSignature((Class) fn.getClass());
+DoFnSignature signature = DoFnSignatures.INSTANCE.getSignature((Class) 
fn.getClass());
 if (signature.processElement().usesSingleWindow()) {
   return new WindowDoFnAdapter<>(fn);
 } else {

Build failed in Jenkins: beam_PostCommit_MavenVerify #1533

[Apache Jenkins Server]

See 

Changes:

[dhalperi] Do not modify the context in NullableCoder

--
[EnvInject] - Mask passwords passed as build parameters.
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 2584ebeb8c57314288a5eba480a8bd2c7dc1b1d2 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2584ebeb8c57314288a5eba480a8bd2c7dc1b1d2
 > git rev-list b7b68e6fb1aafb6b4160e5dcea022bf6c802e33f # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Modules changed, recalculating dependency graph
Established TCP socket on 39758
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_MavenVerify] $ /home/jenkins/tools/java/latest1.8/bin/java 
-Xmx2g -Xms256m -XX:MaxPermSize=512m -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 39758
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 
 
-Dmaven.repo.local=
 -B -e -P release -DrepoToken= clean install coveralls:report 
-DskipITs=false -DintegrationTestPipelineOptions=[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Beam :: Parent
[INFO] Apache Beam :: SDKs
[INFO] Apache Beam :: SDKs :: Java
[INFO] Apache Beam :: SDKs :: Java :: Build Tools
[INFO] Apache Beam :: SDKs :: Java :: Core
[INFO] Apache Beam :: Runners
[INFO] Apache Beam :: Runners :: Core Java
[INFO] Apache Beam :: Runners :: Direct Java
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow
[INFO] Apache Beam :: SDKs :: Java :: IO
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC
[INFO] Apache Beam :: SDKs :: Java :: Extensions
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
[INFO] Apache Beam :: SDKs :: Java :: Microbenchmarks
[INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests
[INFO] Apache Beam :: Runners :: Flink
[INFO] Apache Beam :: Runners :: Flink :: Core
[INFO] Apache Beam :: Runners :: Flink :: Examples
[INFO] Apache Beam :: Runners :: Spark
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
[INFO] Apache Beam :: Examples
[INFO] Apache Beam :: Examples :: Java
[INFO] Apache Beam :: Examples :: Java 8
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
no dependency information available
[WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 

[2/2] incubator-beam git commit: This closes #1103

[kenn]

This closes #1103


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/87c5839d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/87c5839d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/87c5839d

Branch: refs/heads/master
Commit: 87c5839dc859bfd9f404cee9af6eacf01ffb7686
Parents: 2584ebe 546a572
Author: Kenneth Knowles 
Authored: Fri Oct 14 10:34:37 2016 -0700
Committer: Kenneth Knowles 
Committed: Fri Oct 14 10:34:37 2016 -0700

--
 .../beam/runners/core/SplittableParDo.java  |  2 +-
 .../runners/direct/ParDoOverrideFactory.java|  2 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |  2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  4 +--
 .../sdk/transforms/reflect/DoFnInvokers.java| 12 ---
 .../sdk/transforms/reflect/DoFnSignatures.java  |  6 ++--
 .../DoFnSignaturesProcessElementTest.java   | 12 +++
 .../DoFnSignaturesSplittableDoFnTest.java   | 34 ++--
 .../transforms/reflect/DoFnSignaturesTest.java  | 16 -
 9 files changed, 46 insertions(+), 44 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_MavenVerify » Apache Beam :: Parent #1533

[Apache Jenkins Server]

See 


--
Established TCP socket on 39758
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e -P release -DrepoToken=YZWbqQQnANEGMBgOFvXZEOr7etawgMcOQ clean install 
coveralls:report -DskipITs=false -DintegrationTestPipelineOptions=[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Beam :: Parent
[INFO] Apache Beam :: SDKs
[INFO] Apache Beam :: SDKs :: Java
[INFO] Apache Beam :: SDKs :: Java :: Build Tools
[INFO] Apache Beam :: SDKs :: Java :: Core
[INFO] Apache Beam :: Runners
[INFO] Apache Beam :: Runners :: Core Java
[INFO] Apache Beam :: Runners :: Direct Java
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow
[INFO] Apache Beam :: SDKs :: Java :: IO
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC
[INFO] Apache Beam :: SDKs :: Java :: Extensions
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
[INFO] Apache Beam :: SDKs :: Java :: Microbenchmarks
[INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests
[INFO] Apache Beam :: Runners :: Flink
[INFO] Apache Beam :: Runners :: Flink :: Core
[INFO] Apache Beam :: Runners :: Flink :: Examples
[INFO] Apache Beam :: Runners :: Spark
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
[INFO] Apache Beam :: Examples
[INFO] Apache Beam :: Examples :: Java
[INFO] Apache Beam :: Examples :: Java 8
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
no dependency information available
[WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Parent 0.3.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ beam-parent ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @ beam-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ beam-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
beam-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ beam-parent ---
[INFO] Skipping pom project
[INFO] 
[INFO] --- apache-rat-plugin:0.11:check (default) @ beam-parent ---
[INFO] 51 implicit excludes (use -debug for more details).
[INFO] Exclude: **/target/**/*
[INFO] Exclude: **/dependency-reduced-pom.xml
[INFO] Exclude: .github/**/*
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/package-list
[INFO] Exclude: **/user.avsc
[INFO] Exclude: **/test/resources/**/*.txt
[INFO] Exclude: **/test/**/.placeholder
[INFO] Exclude: .repository/**/*
[INFO] Exclude: **/.checkstyle
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.factorypath
[INFO] Exclude: **/.project
[INFO] Exclude: **/.settings/**/*
[INFO] 1322 resources included (use -debug for more details)
[INFO] Rat check: Summary of files. Unapproved: 1 unknown: 1 generated: 0 
approved: 1317 licence.

[GitHub] incubator-beam pull request #1103: Remove getOrDoThisOtherThing pattern from...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1103


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-469) NullableCoder should encode using the context given and not always use the nested context

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-469:
-
Assignee: Thomas Groh

> NullableCoder should encode using the context given and not always use the 
> nested context
> -
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable

[Andrew Martin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Martin closed BEAM-741.
--
Resolution: Not A Problem

> Values transform does not use the correct output coder when values is an 
> Iterable
> 
>
> Key: BEAM-741
> URL: https://issues.apache.org/jira/browse/BEAM-741
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Andrew Martin
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable

[Andrew Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575937#comment-15575937
 ] 

Andrew Martin commented on BEAM-741:


[~kenn] I dug into this more today, and found the specific reason for this 
failure - the inference process in Beam checks the coder registry, and if it 
doesn't find any will try to use a fallback coder provider. If it fails there, 
only then will it try to obtain the coder from the producing transform. In Scio 
we set our own fallback coder provider, so Beam will never end up using the 
output coder from the producing transform. So, in Scio we probably need to 
prefer using the default output coder of the producing transform, and fall back 
as a last resort. I will close this because it is an issue in Scio, not in Beam.

Thanks!

> Values transform does not use the correct output coder when values is an 
> Iterable
> 
>
> Key: BEAM-741
> URL: https://issues.apache.org/jira/browse/BEAM-741
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Andrew Martin
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder should encode using the context given and not always use the nested context

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575934#comment-15575934
 ] 

ASF GitHub Bot commented on BEAM-469:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/992


> NullableCoder should encode using the context given and not always use the 
> nested context
> -
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Priority: Trivial
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #992: [BEAM-469] Do not modify the context in Nu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/992


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Do not modify the context in NullableCoder

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b7b68e6fb -> 2584ebeb8


Do not modify the context in NullableCoder

The NullableCoder does not encode any elements after the subcoder
encodes the input value for non-null values. As a result, the subcoder
should see the entire input stream if it is available.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2c056b17
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2c056b17
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2c056b17

Branch: refs/heads/master
Commit: 2c056b171b6e329ca5c025eb6fbc81cff29a8950
Parents: b7b68e6
Author: Thomas Groh 
Authored: Fri Sep 23 08:58:07 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 10:27:15 2016 -0700

--
 .../apache/beam/sdk/coders/NullableCoder.java   |  4 +-
 .../beam/sdk/coders/NullableCoderTest.java  | 49 +---
 2 files changed, 44 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c056b17/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
index 9c6c7c0..29b697c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
@@ -79,7 +79,7 @@ public class NullableCoder extends StandardCoder {
   outStream.write(ENCODE_NULL);
 } else {
   outStream.write(ENCODE_PRESENT);
-  valueCoder.encode(value, outStream, context.nested());
+  valueCoder.encode(value, outStream, context);
 }
   }
 
@@ -94,7 +94,7 @@ public class NullableCoder extends StandardCoder {
 "NullableCoder expects either a byte valued %s (null) or %s 
(present), got %s",
 ENCODE_NULL, ENCODE_PRESENT, b));
 }
-return valueCoder.decode(inStream, context.nested());
+return valueCoder.decode(inStream, context);
   }
 
   @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c056b17/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
index 61e7e41..425d5ba 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.coders;
 
+import static com.google.common.base.Preconditions.checkArgument;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.theInstance;
 import static org.junit.Assert.assertEquals;
@@ -26,9 +27,13 @@ import static org.junit.Assert.assertTrue;
 
 import com.google.common.collect.ImmutableList;
 import java.io.ByteArrayInputStream;
+import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
+import org.apache.beam.sdk.coders.Coder.Context;
 import org.apache.beam.sdk.testing.CoderProperties;
 import org.junit.Rule;
 import org.junit.Test;
@@ -76,14 +81,15 @@ public class NullableCoderTest {
* @see org.apache.beam.sdk.coders.PrintBase64Encodings
*/
   private static final List TEST_ENCODINGS = Arrays.asList(
-  "AQA",
-  "AQFh",
-  "AQIxMw",
-  "AQVoZWxsbw",
+  "AQ",
+  "AWE",
+  "ATEz",
+  "AWhlbGxv",
   "AA",
-  "AShhIGxvbmdlciBzdHJpbmcgd2l0aCBzcGFjZXMgYW5kIGFsbCB0aGF0",
-  "ARlhIHN0cmluZyB3aXRoIGEgCiBuZXdsaW5l",
-  "AQ_jgrnjgr_jg6rjg7PjgrA");
+  "AWEgbG9uZ2VyIHN0cmluZyB3aXRoIHNwYWNlcyBhbmQgYWxsIHRoYXQ",
+  "AWEgc3RyaW5nIHdpdGggYSAKIG5ld2xpbmU",
+  "AeOCueOCv-ODquODs-OCsA"
+  );
 
   @Test
   public void testWireFormatEncode() throws Exception {
@@ -135,8 +141,37 @@ public class NullableCoderTest {
   }
 
   @Test
+  public void testSubcoderRecievesEntireStream() throws Exception {
+NullableCoder coder = NullableCoder.of(new 
EntireStreamExpectingCoder());
+
+CoderProperties.coderDecodeEncodeEqualInContext(coder, Context.OUTER, 
null);
+CoderProperties.coderDecodeEncodeEqualInContext(coder, Context.OUTER, 
"foo");
+  }
+
+  @Test
   public void testNestedNullableCoder() {
 NullableCoder coder = NullableCoder.of(DoubleCoder.of());
 assertThat(NullableCoder.of(coder), 

[2/2] incubator-beam git commit: Closes #992

[dhalperi]

Closes #992


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2584ebeb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2584ebeb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2584ebeb

Branch: refs/heads/master
Commit: 2584ebeb8c57314288a5eba480a8bd2c7dc1b1d2
Parents: b7b68e6 2c056b1
Author: Dan Halperin 
Authored: Fri Oct 14 10:27:16 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 10:27:16 2016 -0700

--
 .../apache/beam/sdk/coders/NullableCoder.java   |  4 +-
 .../beam/sdk/coders/NullableCoderTest.java  | 49 +---
 2 files changed, 44 insertions(+), 9 deletions(-)
--

[jira] [Commented] (BEAM-55) Allow users to compress FileBasedSink output files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575905#comment-15575905
 ] 

ASF GitHub Bot commented on BEAM-55:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1027


> Allow users to compress FileBasedSink output files
> --
>
> Key: BEAM-55
> URL: https://issues.apache.org/jira/browse/BEAM-55
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Priority: Minor
>
> FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option 
> for compressing its output.
> In general, we discourage compression because it limits or blocks scalably 
> reading from a file in parallel. However, users may want it -- so we should 
> support the option (with appropriate warnings).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-55] TextIO & FileBasedSink: Add support for compressed output

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 49f944430 -> b7b68e6fb


[BEAM-55] TextIO & FileBasedSink: Add support for compressed output

[BEAM-55] Add customizable file-based output support through DecoratedFileSink
and concrete Gzip file-based output support through
WriterOutputGzipDecoratorFactory
[BEAM-55] Add example pipeline usage and a few eclipse-related entries to
gitignore
[BEAM-55] Move logic from DecoratedFileSink into FileBasedSink and TextIO to
allow for direct support of customizable file-based, including built-in Gzip
and Bzip2 support, per request by @dhalperi
[BEAM-697] TextIO.Write.Bound.withHeader() and withFooter() now pass
current value of validate to returned Bound instance instead of literal 'false'
[BEAM-55] Simplified overlycomplex API, removed example pipeline, cleaned up
comment formatting, added small usage example to main TextIO javadoc,
per request by @dhalperi
[BEAM-55] Revert erroneous formatting, remove old comments, fix javadoc, inline
superclass only used in one unit test
[BEAM-55] Add writableByteChannelFactory to DisplayData, move fix for BEAM-697 
into its own PR


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fa589ee4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fa589ee4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fa589ee4

Branch: refs/heads/master
Commit: fa589ee4e16b91a06e2b78d870b1fa70ba5834e8
Parents: 49f9444
Author: Jeffrey Scott Keone Payne 
Authored: Thu Sep 29 12:23:44 2016 -0700
Committer: Dan Halperin 
Committed: Fri Oct 14 10:14:48 2016 -0700

--
 .gitignore  |   5 +
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 138 ++-
 .../java/org/apache/beam/sdk/io/TextIO.java |  91 +---
 .../sdk/io/DrunkWritableByteChannelFactory.java |  80 +++
 .../apache/beam/sdk/io/FileBasedSinkTest.java   | 108 +++
 .../java/org/apache/beam/sdk/io/TextIOTest.java |  59 ++--
 6 files changed, 447 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa589ee4/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 5390dd0..d6cffec 100644
--- a/.gitignore
+++ b/.gitignore
@@ -17,6 +17,8 @@ bin/
 .project
 .factorypath
 .checkstyle
+.fbExcludeFilterFile
+.apt_generated/
 .settings/
 
 # The build process generates the dependency-reduced POM, but it shouldn't be
@@ -27,6 +29,9 @@ dependency-reduced-pom.xml
 # produced by a text editor.
 *~
 
+# Ignore MacOSX files.
+.DS_Store
+
 # NOTE: if you modify this file, you probably need to modify the file set that
 # is an input to 'maven-assembly-plugin' that generates source distribution.
 # This is typically in files named 'src.xml' throughout this repository.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa589ee4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index f571d50..7e95c5b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -20,11 +20,14 @@ package org.apache.beam.sdk.io;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
+import static com.google.common.base.Strings.isNullOrEmpty;
 
 import com.google.common.collect.Ordering;
+
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
+import java.nio.channels.Channels;
 import java.nio.channels.WritableByteChannel;
 import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
@@ -34,6 +37,10 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.HashSet;
 import java.util.List;
+import java.util.zip.GZIPOutputStream;
+
+import javax.annotation.Nullable;
+
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.SerializableCoder;
 import org.apache.beam.sdk.options.PipelineOptions;
@@ -45,6 +52,7 @@ import org.apache.beam.sdk.util.GcsUtil.GcsUtilFactory;
 import org.apache.beam.sdk.util.IOChannelFactory;
 import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.util.MimeTypes;
+import 
org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -69,6 +77,64 @@ 

[GitHub] incubator-beam pull request #1103: Remove "OrParse" from DoFnSignatures.getS...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1103

Remove "OrParse" from DoFnSignatures.getSignature

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam getOrNothing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1103.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1103


commit c07f91a3f0a722ded5a667acfba3fdf2dce155cb
Author: Kenneth Knowles 
Date:   2016-10-14T15:48:49Z

Remove "OrParse" from DoFnSignatures.getSignature

This excessive verbosity is also an abstraction violation. It
is an implementation detail how the signature is gotten.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-638) Add a Window function to create a bounded PCollection from an unbounded one

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575679#comment-15575679
 ] 

Eugene Kirpichov commented on BEAM-638:
---

More discussion currently happening on 
http://markmail.org/message/se23dgiymob2pgok

> Add a Window function to create a bounded PCollection from an unbounded one
> ---
>
> Key: BEAM-638
> URL: https://issues.apache.org/jira/browse/BEAM-638
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Jean-Baptiste Onofré
>Assignee: Davor Bonaci
>
> Today, if the pipeline source is unbounded, and the sink expects a bounded 
> collection, there's no way to use a single pipeline. Even a window creates a 
> chunk on the unbounded PCollection, but the "sub" PCollection is still 
> unbounded.
> It would be helpful for users to have a Window function that create a bounded 
> PCollection (on the window) from an unbounded PCollection coming from the 
> source.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1334

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1531

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1531

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1531

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-735) PAssertStreaming should make sure the assertion happened.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-735.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> PAssertStreaming should make sure the assertion happened.
> -
>
> Key: BEAM-735
> URL: https://issues.apache.org/jira/browse/BEAM-735
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> The Spark runner currently runs PAsserts via `PAssertStreaming` which groups 
> into a single key and asserts the values on the worker (part of the "Lambda" 
> in the Spark lingo).
> This could be a problem since Spark won't run if there's nothing to process - 
> so that if for some reason the input is missed, say reading from Kafka latest 
> or simply an empty topic, the assertion will be skipped and so we'll never 
> fail (we would like to fail if there was no input, while we expected one).
> This might change once Spark provide a better support for the Beam model in 
> streaming, but until then, it's best that our tests will consider this case 
> as well.
> I'll add an aggregator and increment for assertion, at the end I'll make sure 
> the aggregator is not 0, so that at least one assertion took place (if for 
> some reason Spark kept on for a couple of more intervals it might execute the 
> same assertion more then once, if the input is repeated).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-735) PAssertStreaming should make sure the assertion happened.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575453#comment-15575453
 ] 

ASF GitHub Bot commented on BEAM-735:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1073


> PAssertStreaming should make sure the assertion happened.
> -
>
> Key: BEAM-735
> URL: https://issues.apache.org/jira/browse/BEAM-735
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> The Spark runner currently runs PAsserts via `PAssertStreaming` which groups 
> into a single key and asserts the values on the worker (part of the "Lambda" 
> in the Spark lingo).
> This could be a problem since Spark won't run if there's nothing to process - 
> so that if for some reason the input is missed, say reading from Kafka latest 
> or simply an empty topic, the assertion will be skipped and so we'll never 
> fail (we would like to fail if there was no input, while we expected one).
> This might change once Spark provide a better support for the Beam model in 
> streaming, but until then, it's best that our tests will consider this case 
> as well.
> I'll add an aggregator and increment for assertion, at the end I'll make sure 
> the aggregator is not 0, so that at least one assertion took place (if for 
> some reason Spark kept on for a couple of more intervals it might execute the 
> same assertion more then once, if the input is repeated).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1073

[amitsela]

This closes #1073


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/49f94443
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/49f94443
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/49f94443

Branch: refs/heads/master
Commit: 49f94443004a48c0c1524f3c431b73b0f94d53a2
Parents: d790dfe 44225cf
Author: Sela 
Authored: Fri Oct 14 16:54:54 2016 +0300
Committer: Sela 
Committed: Fri Oct 14 16:54:54 2016 +0300

--
 .../beam/runners/spark/EvaluationResult.java|   4 +-
 .../spark/translation/EvaluationContext.java|   3 +-
 .../streaming/StreamingEvaluationContext.java   |   6 +-
 .../apache/beam/runners/spark/DeDupTest.java|   3 +-
 .../beam/runners/spark/EmptyInputTest.java  |   1 -
 .../beam/runners/spark/SimpleWordCountTest.java |   6 +-
 .../apache/beam/runners/spark/TfIdfTest.java|   3 +-
 .../beam/runners/spark/io/AvroPipelineTest.java |   4 +-
 .../beam/runners/spark/io/NumShardsTest.java|   4 +-
 .../io/hadoop/HadoopFileFormatPipelineTest.java |   4 +-
 .../spark/translation/CombineGloballyTest.java  |   1 -
 .../spark/translation/CombinePerKeyTest.java|   1 -
 .../spark/translation/DoFnOutputTest.java   |   4 +-
 .../translation/MultiOutputWordCountTest.java   |   2 -
 .../spark/translation/SerializationTest.java|   4 +-
 .../translation/WindowedWordCountTest.java  |  10 +-
 .../streaming/EmptyStreamAssertionTest.java |  76 
 .../streaming/FlattenStreamingTest.java |  11 +-
 .../streaming/KafkaStreamingTest.java   |   6 +-
 .../RecoverFromCheckpointStreamingTest.java | 179 --
 .../ResumeFromCheckpointStreamingTest.java  | 182 +++
 .../streaming/SimpleStreamingWordCountTest.java |   6 +-
 .../streaming/utils/PAssertStreaming.java   |  87 ++---
 23 files changed, 346 insertions(+), 261 deletions(-)
--

[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[Jeffrey Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575414#comment-15575414
 ] 

Jeffrey Payne commented on BEAM-697:


Will do!

> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575371#comment-15575371
 ] 

Maximilian Michels commented on BEAM-593:
-

Until this is probably addressed on the Flink side, let us return 
{{State.DONE}} for all pipeline results which are created in blocking execution 
mode. For detached executions, we should always return {{State.UNKNOWN}} 
because we don't have a means yet to access the Job state.

This will change once Flink has introduced a client for job control in the API.

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #45: Fix jekyll setup instructions

[cschneider]

GitHub user cschneider opened a pull request:

https://github.com/apache/incubator-beam-site/pull/45

Fix jekyll setup instructions

When I just tried the jekyll setup on a recent ubuntu the start of jekyll 
failed with an error.
It reported that it needs an activesupport < 5.0.0 while the version the 
listed commands installed was 5.0.1.

What I found is the workaround below. I guess this needs to be fixed in the 
jekyll gems too.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cschneider/incubator-beam-site 
Fix-jekyll-setup-instructions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/45.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #45


commit 536af368dd722f343b98ed72ee40a54f8b5483c3
Author: Christian Schneider 
Date:   2016-10-14T13:08:25Z

Fix jekyll setup instructions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-748) Issues with local processes in MongoDbIO and JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575269#comment-15575269
 ] 

Jean-Baptiste Onofré commented on BEAM-748:
---

By the way, it could also be caused by no space left on filesystem.

> Issues with local processes in MongoDbIO and JdbcIO
> ---
>
> Key: BEAM-748
> URL: https://issues.apache.org/jira/browse/BEAM-748
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Hey JB, Can you take a look at these issues? Causing flakes in precommit 
> testing.
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3931/
> A few different types, likely due to issues setting up databases and reusing 
> them across tests?
> {code}
> java.net.ConnectException : Error connecting to server localhost on port 
> 1,527 with message Connection refused.
> {code}
> {code}
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
> Stacktrace
> java.sql.SQLTransactionRollbackException: 
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
>   at 
> org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
>   at org.apache.derby.client.am.SqlException.getSQLException(Unknown 
> Source)
>   at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown 
> Source)
>   at org.apache.beam.sdk.io.jdbc.JdbcIOTest.initTable(JdbcIOTest.java:106)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>   at 
> 

[GitHub] incubator-beam-site pull request #44: Add IDE setup instructions

[cschneider]

GitHub user cschneider opened a pull request:

https://github.com/apache/incubator-beam-site/pull/44

Add IDE setup instructions

See my mail "Documentation for IDE setup" on the dev list for details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cschneider/incubator-beam-site ide-setup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44


commit 379375e558b018d4495ba626833ec5d180cf3477
Author: Christian Schneider 
Date:   2016-10-14T12:56:01Z

Add IDE setup instructions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1530

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1530

[Apache Jenkins Server]

See