[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528505#comment-15528505
 ] 

Jean-Baptiste Onofré commented on BEAM-679:
---

Cool, thanks, just saw that this morning.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Luke Cwik
>Priority: Critical
> Fix For: 0.3.0-incubating
>
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1223

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1222

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-604) Use Watermark Check Streaming Job Finish in TestDataflowRunner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-604.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Use Watermark Check Streaming Job Finish in TestDataflowRunner
> --
>
> Key: BEAM-604
> URL: https://issues.apache.org/jira/browse/BEAM-604
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Currently, streaming job with bounded input can't be terminated automatically 
> and TestDataflowRunner can't handle this case. Need to update 
> TestDataflowRunner so that streaming integration test such as 
> WindowedWordCountIT can run with it.
> Implementation:
> Query watermark of each step and wait until all watermarks set to MAX then 
> cancel the job.
> Update:
> Suggesting by [~pei...@gmail.com], implement checkMaxWatermark in 
> DataflowPipelineJob#waitUntilFinish. Thus, all dataflow streaming jobs with 
> bounded input will take advantage of this change and are canceled 
> automatically when watermarks reach to max value. Also Dataflow runners can 
> keep simple and free from handling batch and streaming two cases.
> Update:
> Pipeline author should have control on whether or not canceling streaming job 
> and when. Test framework is a better place to auto-cancel streaming test job 
> when curtain conditions meet, rather than in waitUntilFinish().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/5] incubator-beam git commit: [BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

[lcwik]

[BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f6bd47ba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f6bd47ba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f6bd47ba

Branch: refs/heads/master
Commit: f6bd47ba79615d90660b4105d8a6c5c276af8551
Parents: 2571cfc
Author: Mark Liu 
Authored: Fri Sep 2 14:41:11 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 17:03:56 2016 -0700

--
 .../beam/examples/WindowedWordCountIT.java  |  49 ++
 .../dataflow/testing/TestDataflowRunner.java|  52 +-
 .../testing/TestDataflowRunnerTest.java | 158 +++
 3 files changed, 255 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f6bd47ba/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
new file mode 100644
index 000..890ca2b
--- /dev/null
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.examples;
+
+import java.io.IOException;
+import org.apache.beam.examples.WindowedWordCount.Options;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * End-to-end integration test of {@link WindowedWordCount}.
+ */
+@RunWith(JUnit4.class)
+public class WindowedWordCountIT {
+
+  /**
+   * Options for the {@link WindowedWordCount} Integration Test.
+   */
+  public interface TestOptions extends Options, TestPipelineOptions {
+  }
+
+  @Test
+  public void testWindowedWordCount() throws IOException {
+PipelineOptionsFactory.register(TestOptions.class);
+TestOptions options = 
TestPipeline.testingPipelineOptions().as(TestOptions.class);
+
+WindowedWordCount.main(TestPipeline.convertToArgs(options));
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f6bd47ba/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
index 9be773b..c569cd4 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
@@ -58,6 +58,8 @@ import org.slf4j.LoggerFactory;
  */
 public class TestDataflowRunner extends PipelineRunner {
   private static final String TENTATIVE_COUNTER = "tentative";
+  private static final String WATERMARK_METRIC_SUFFIX = 
"windmill-data-watermark";
+  private static final long MAX_WATERMARK_VALUE = -2L;
   private static final Logger LOG = 
LoggerFactory.getLogger(TestDataflowRunner.class);
 
   private final TestDataflowPipelineOptions options;
@@ -121,11 +123,17 @@ public class TestDataflowRunner extends 
PipelineRunner {
 if (result.isPresent()) {
   return result;
 }
+result = checkMaxWatermark(job);
+if (result.isPresent()) {
+  return result;
+}
 Thread.sleep(1L);
   }
   

[1/5] incubator-beam git commit: Add StreamingIT category to integration test

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2571cfcf5 -> db47c63ab


Add StreamingIT category to integration test


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fbae96f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fbae96f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fbae96f2

Branch: refs/heads/master
Commit: fbae96f208afe3dca3ef620c4a312345a6314e13
Parents: 9f0588a
Author: Mark Liu 
Authored: Thu Sep 8 10:52:36 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 17:03:56 2016 -0700

--
 examples/java/pom.xml   | 31 
 .../beam/examples/WindowedWordCountIT.java  | 18 ++--
 .../apache/beam/sdk/testing/StreamingIT.java| 24 +++
 3 files changed, 71 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fbae96f2/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 9a48ec6..31244db 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -216,6 +216,37 @@
 
   
 
+
+
+
+  disable-streaming-ITs
+  false
+  
+
+  
+org.apache.maven.plugins
+maven-failsafe-plugin
+
+  
+
+  integration-test
+  verify
+
+
+  
org.apache.beam.sdk.testing.StreamingIT
+
+  
+
+  
+
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fbae96f2/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index 890ca2b..cddcd4a 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -21,9 +21,12 @@ package org.apache.beam.examples;
 import java.io.IOException;
 import org.apache.beam.examples.WindowedWordCount.Options;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.testing.StreamingIT;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
 
@@ -36,13 +39,24 @@ public class WindowedWordCountIT {
   /**
* Options for the {@link WindowedWordCount} Integration Test.
*/
-  public interface TestOptions extends Options, TestPipelineOptions {
+  public interface TestOptions extends Options, TestPipelineOptions, 
StreamingOptions{
   }
 
   @Test
-  public void testWindowedWordCount() throws IOException {
+  public void testWindowedWordCountInBatch() throws IOException {
+testWindowedWordCountPipeline(false);
+  }
+
+  @Test
+  @Category(StreamingIT.class)
+  public void testWindowedWordCountInStreaming() throws IOException {
+testWindowedWordCountPipeline(true);
+  }
+
+  private void testWindowedWordCountPipeline(boolean isStreaming) throws 
IOException {
 PipelineOptionsFactory.register(TestOptions.class);
 TestOptions options = 
TestPipeline.testingPipelineOptions().as(TestOptions.class);
+options.setStreaming(isStreaming);
 
 WindowedWordCount.main(TestPipeline.convertToArgs(options));
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fbae96f2/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StreamingIT.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StreamingIT.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StreamingIT.java
new file mode 100644
index 000..b3dd4a0
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StreamingIT.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the 

[4/5] incubator-beam git commit: More javadoc and keep retry in case of get metrics exception

[lcwik]

More javadoc and keep retry in case of get metrics exception


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/370c1714
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/370c1714
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/370c1714

Branch: refs/heads/master
Commit: 370c171450215d3fac4208875850279a796415c9
Parents: fbae96f
Author: Mark Liu 
Authored: Wed Sep 14 13:18:40 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 17:03:57 2016 -0700

--
 examples/java/pom.xml   |   7 +-
 .../beam/examples/WindowedWordCountIT.java  |  13 +-
 .../dataflow/testing/TestDataflowRunner.java| 158 +-
 .../testing/TestDataflowRunnerTest.java | 287 ++-
 .../apache/beam/sdk/testing/StreamingIT.java|  13 +-
 5 files changed, 263 insertions(+), 215 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/370c1714/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 31244db..6b1b7ce 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -218,11 +218,8 @@
 
 
 
 
   disable-streaming-ITs

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/370c1714/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index cddcd4a..379d1b0 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -15,7 +15,6 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-
 package org.apache.beam.examples;
 
 import java.io.IOException;
@@ -39,23 +38,25 @@ public class WindowedWordCountIT {
   /**
* Options for the {@link WindowedWordCount} Integration Test.
*/
-  public interface TestOptions extends Options, TestPipelineOptions, 
StreamingOptions{
+  public interface WindowedWordCountITOptions
+  extends Options, TestPipelineOptions, StreamingOptions {
   }
 
   @Test
   public void testWindowedWordCountInBatch() throws IOException {
-testWindowedWordCountPipeline(false);
+testWindowedWordCountPipeline(false /* isStreaming */);
   }
 
   @Test
   @Category(StreamingIT.class)
   public void testWindowedWordCountInStreaming() throws IOException {
-testWindowedWordCountPipeline(true);
+testWindowedWordCountPipeline(true /* isStreaming */);
   }
 
   private void testWindowedWordCountPipeline(boolean isStreaming) throws 
IOException {
-PipelineOptionsFactory.register(TestOptions.class);
-TestOptions options = 
TestPipeline.testingPipelineOptions().as(TestOptions.class);
+PipelineOptionsFactory.register(WindowedWordCountITOptions.class);
+WindowedWordCountITOptions options =
+
TestPipeline.testingPipelineOptions().as(WindowedWordCountITOptions.class);
 options.setStreaming(isStreaming);
 
 WindowedWordCount.main(TestPipeline.convertToArgs(options));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/370c1714/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
index b8b4eaf..a152505 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
@@ -22,6 +22,7 @@ import static org.hamcrest.MatcherAssert.assertThat;
 import com.google.api.services.dataflow.model.JobMessage;
 import com.google.api.services.dataflow.model.JobMetrics;
 import com.google.api.services.dataflow.model.MetricUpdate;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Joiner;
 import com.google.common.base.Optional;
 import com.google.common.base.Throwables;
@@ -31,6 +32,7 @@ import java.util.List;
 import java.util.concurrent.Callable;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
+import 

[GitHub] incubator-beam pull request #916: [BEAM-604] Use Watermark to Finish Streami...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/916


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/5] incubator-beam git commit: Fix checkMaxWatermark causing batch test failed

[lcwik]

Fix checkMaxWatermark causing batch test failed


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f0588a2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9f0588a2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9f0588a2

Branch: refs/heads/master
Commit: 9f0588a2d63dd8538675b128a488ea5fa9c491f2
Parents: f6bd47b
Author: Mark Liu 
Authored: Wed Sep 7 11:59:02 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 17:03:56 2016 -0700

--
 .../dataflow/testing/TestDataflowRunner.java| 20 +++---
 .../testing/TestDataflowRunnerTest.java | 40 ++--
 2 files changed, 33 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f0588a2/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
index c569cd4..b8b4eaf 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunner.java
@@ -120,11 +120,7 @@ public class TestDataflowRunner extends 
PipelineRunner {
 try {
   for (;;) {
 Optional result = checkForSuccess(job);
-if (result.isPresent()) {
-  return result;
-}
-result = checkMaxWatermark(job);
-if (result.isPresent()) {
+if (result.isPresent() && (!result.get() || 
checkMaxWatermark(job))) {
   return result;
 }
 Thread.sleep(1L);
@@ -217,7 +213,7 @@ public class TestDataflowRunner extends 
PipelineRunner {
 + "{} expected assertions.", job.getJobId(), successes, failures,
 expectedNumberOfAssertions);
 return Optional.of(false);
-  } else if (successes > 0 && successes >= expectedNumberOfAssertions) {
+  } else if (successes >= expectedNumberOfAssertions) {
 LOG.info("Found result while running Dataflow job {}. Found {} 
success, {} failures out of "
 + "{} expected assertions.", job.getJobId(), successes, failures,
 expectedNumberOfAssertions);
@@ -231,13 +227,7 @@ public class TestDataflowRunner extends 
PipelineRunner {
 return Optional.absent();
   }
 
-  Optional checkMaxWatermark(DataflowPipelineJob job) throws 
IOException {
-State state = job.getState();
-if (state == State.FAILED || state == State.CANCELLED) {
-  LOG.info("The pipeline {}", state);
-  return Optional.of(false);
-}
-
+  boolean checkMaxWatermark(DataflowPipelineJob job) throws IOException {
 JobMetrics metrics = options.getDataflowClient().projects().jobs()
 .getMetrics(job.getProjectId(), job.getJobId()).execute();
 
@@ -260,10 +250,10 @@ public class TestDataflowRunner extends 
PipelineRunner {
   }
   if (hasMaxWatermark) {
 LOG.info("All watermarks of job {} reach to max value.", 
job.getJobId());
-return Optional.of(true);
+return true;
   }
 }
-return Optional.absent();
+return false;
   }
 
   @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f0588a2/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunnerTest.java
index 70c4562..3818b35 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunnerTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowRunnerTest.java
@@ -19,8 +19,10 @@ package org.apache.beam.runners.dataflow.testing;
 
 import static org.hamcrest.Matchers.containsString;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertSame;
 import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
 import static 

[jira] [Commented] (BEAM-604) Use Watermark Check Streaming Job Finish in TestDataflowRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527868#comment-15527868
 ] 

ASF GitHub Bot commented on BEAM-604:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/916


> Use Watermark Check Streaming Job Finish in TestDataflowRunner
> --
>
> Key: BEAM-604
> URL: https://issues.apache.org/jira/browse/BEAM-604
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
>
> Currently, streaming job with bounded input can't be terminated automatically 
> and TestDataflowRunner can't handle this case. Need to update 
> TestDataflowRunner so that streaming integration test such as 
> WindowedWordCountIT can run with it.
> Implementation:
> Query watermark of each step and wait until all watermarks set to MAX then 
> cancel the job.
> Update:
> Suggesting by [~pei...@gmail.com], implement checkMaxWatermark in 
> DataflowPipelineJob#waitUntilFinish. Thus, all dataflow streaming jobs with 
> bounded input will take advantage of this change and are canceled 
> automatically when watermarks reach to max value. Also Dataflow runners can 
> keep simple and free from handling batch and streaming two cases.
> Update:
> Pipeline author should have control on whether or not canceling streaming job 
> and when. Test framework is a better place to auto-cancel streaming test job 
> when curtain conditions meet, rather than in waitUntilFinish().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/5] incubator-beam git commit: [BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

[lcwik]

[BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

This closes #916


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/db47c63a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/db47c63a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/db47c63a

Branch: refs/heads/master
Commit: db47c63abe73fce979529ee9e16721eee4d649d7
Parents: 2571cfc 370c171
Author: Luke Cwik 
Authored: Tue Sep 27 17:04:27 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 17:04:27 2016 -0700

--
 examples/java/pom.xml   |  28 ++
 .../beam/examples/WindowedWordCountIT.java  |  64 +
 .../dataflow/testing/TestDataflowRunner.java| 144 +++---
 .../testing/TestDataflowRunnerTest.java | 287 +++
 .../apache/beam/sdk/testing/StreamingIT.java|  35 +++
 5 files changed, 466 insertions(+), 92 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1221

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-559) DoFnTester should handle Setup / TearDown

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527667#comment-15527667
 ] 

Eugene Kirpichov commented on BEAM-559:
---

This issue should be closed now.

> DoFnTester should handle Setup / TearDown
> -
>
> Key: BEAM-559
> URL: https://issues.apache.org/jira/browse/BEAM-559
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Eugene Kirpichov
>
> Now that DoFn supports setup and teardown, it would be nice for DoFnTester to 
> add them to its lifecycle so as to avoid calling these methods explicitly in 
> DoFn unit tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-687) Repeated 'mvn install' fails with "java.lang.IllegalStateException: endPosTable already set"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527566#comment-15527566
 ] 

ASF GitHub Bot commented on BEAM-687:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1020

[BEAM-687] Upgrade maven-compiler-plugin to fix incremental build

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

We are hitting an issue with AutoValue and the current version of
maven-compiler-plugin which breaks incremental build. See:
https://issues.apache.org/jira/browse/MCOMPILER-236

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam autovalue-build-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1020


commit f00fa18eaf127ade24f68b5a15c3a72f39b32a09
Author: Scott Wegner 
Date:   2016-09-27T22:09:46Z

Upgrade maven-compiler-plugin to fix incremental build

We are hitting an issue with AutoValue and the current version of
maven-compiler-plugin which breaks incremental build. See:
https://issues.apache.org/jira/browse/MCOMPILER-236




> Repeated 'mvn install' fails with "java.lang.IllegalStateException: 
> endPosTable already set"
> 
>
> Key: BEAM-687
> URL: https://issues.apache.org/jira/browse/BEAM-687
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> See [mail 
> thread|http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201609.mbox/%3CCAOdmQRgb8_aDUu%3DKt7LgJfLHJwffmZoBOWZ%3D69AcrTXgZXncmA%40mail.gmail.com%3E].
> It appears that we're hitting [this 
> issue|https://issues.apache.org/jira/browse/MCOMPILER-236] with 
> maven-compiler-plugin, AutoValue, and incremental builds. According to the 
> bug thread, the issue is fixed with maven-compiler-plugin version 3.5.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-687) Repeated 'mvn install' fails with "java.lang.IllegalStateException: endPosTable already set"

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-687:
-

 Summary: Repeated 'mvn install' fails with 
"java.lang.IllegalStateException: endPosTable already set"
 Key: BEAM-687
 URL: https://issues.apache.org/jira/browse/BEAM-687
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Scott Wegner
Assignee: Scott Wegner


See [mail 
thread|http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201609.mbox/%3CCAOdmQRgb8_aDUu%3DKt7LgJfLHJwffmZoBOWZ%3D69AcrTXgZXncmA%40mail.gmail.com%3E].

It appears that we're hitting [this 
issue|https://issues.apache.org/jira/browse/MCOMPILER-236] with 
maven-compiler-plugin, AutoValue, and incremental builds. According to the bug 
thread, the issue is fixed with maven-compiler-plugin version 3.5.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-559) DoFnTester should handle Setup / TearDown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527536#comment-15527536
 ] 

ASF GitHub Bot commented on BEAM-559:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1011


> DoFnTester should handle Setup / TearDown
> -
>
> Key: BEAM-559
> URL: https://issues.apache.org/jira/browse/BEAM-559
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Eugene Kirpichov
>
> Now that DoFn supports setup and teardown, it would be nice for DoFnTester to 
> add them to its lifecycle so as to avoid calling these methods explicitly in 
> DoFn unit tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1011: [BEAM-559] Support for @Setup and @Teardo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1011


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Support for @Setup and @Teardown in DoFnTester

[dhalperi]

Support for @Setup and @Teardown in DoFnTester


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bef0e9de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bef0e9de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bef0e9de

Branch: refs/heads/master
Commit: bef0e9de02be051411f20b298168e8477ed1a0da
Parents: 9009802
Author: Eugene Kirpichov 
Authored: Mon Sep 26 16:58:20 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 27 14:57:49 2016 -0700

--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 120 +++--
 .../beam/sdk/transforms/DoFnTesterTest.java | 456 +++
 2 files changed, 338 insertions(+), 238 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bef0e9de/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 0e018ba..9adb806 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -78,10 +78,10 @@ import org.joda.time.Instant;
  * @param  the type of the {@link DoFn}'s (main) input elements
  * @param  the type of the {@link DoFn}'s (main) output elements
  */
-public class DoFnTester {
+public class DoFnTester implements AutoCloseable {
   /**
* Returns a {@code DoFnTester} supporting unit-testing of the given
-   * {@link DoFn}.
+   * {@link DoFn}. By default, uses {@link CloningBehavior#CLONE_ONCE}.
*/
   @SuppressWarnings("unchecked")
   public static  DoFnTester of(DoFn fn) {
@@ -91,6 +91,8 @@ public class DoFnTester {
   /**
* Returns a {@code DoFnTester} supporting unit-testing of the given
* {@link OldDoFn}.
+   *
+   * @see #of(DoFn)
*/
   @SuppressWarnings("unchecked")
public static  DoFnTester
@@ -108,8 +110,11 @@ public class DoFnTester {
* {@link DoFn} takes no side inputs.
*/
   public void setSideInputs(Map> 
sideInputs) {
+checkState(
+state == State.UNINITIALIZED,
+"Can't add side inputs: DoFnTester is already initialized, in state 
%s",
+state);
 this.sideInputs = sideInputs;
-resetState();
   }
 
   /**
@@ -123,6 +128,10 @@ public class DoFnTester {
* that is used.
*/
   public  void setSideInput(PCollectionView sideInput, BoundedWindow 
window, T value) {
+checkState(
+state == State.UNINITIALIZED,
+"Can't add side inputs: DoFnTester is already initialized, in state 
%s",
+state);
 Map windowValues = (Map) 
sideInputs.get(sideInput);
 if (windowValues == null) {
   windowValues = new HashMap<>();
@@ -132,10 +141,24 @@ public class DoFnTester {
   }
 
   /**
-   * Whether or not a {@link DoFnTester} should clone the {@link DoFn} under 
test.
+   * When a {@link DoFnTester} should clone the {@link DoFn} under test and 
how it should manage
+   * the lifecycle of the {@link DoFn}.
*/
   public enum CloningBehavior {
-CLONE,
+/**
+ * Clone the {@link DoFn} and call {@link DoFn.Setup} every time a bundle 
starts; call {@link
+ * DoFn.Teardown} every time a bundle finishes.
+ */
+CLONE_PER_BUNDLE,
+/**
+ * Clone the {@link DoFn} and call {@link DoFn.Setup} on the first access; 
call {@link
+ * DoFn.Teardown} only explicitly.
+ */
+CLONE_ONCE,
+/**
+ * Do not clone the {@link DoFn}; call {@link DoFn.Setup} on the first 
access; call {@link
+ * DoFn.Teardown} only explicitly.
+ */
 DO_NOT_CLONE
   }
 
@@ -143,6 +166,7 @@ public class DoFnTester {
* Instruct this {@link DoFnTester} whether or not to clone the {@link DoFn} 
under test.
*/
   public void setCloningBehavior(CloningBehavior newValue) {
+checkState(state == State.UNINITIALIZED, "Wrong state: %s", state);
 this.cloningBehavior = newValue;
   }
 
@@ -187,11 +211,17 @@ public class DoFnTester {
   /**
* Calls the {@link DoFn.StartBundle} method on the {@link DoFn} under test.
*
-   * If needed, first creates a fresh instance of the {@link DoFn} under 
test.
+   * If needed, first creates a fresh instance of the {@link DoFn} under 
test and calls
+   * {@link DoFn.Setup}.
*/
   public void startBundle() throws 

[1/2] incubator-beam git commit: Closes #1011

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 900980246 -> 2571cfcf5


Closes #1011


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2571cfcf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2571cfcf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2571cfcf

Branch: refs/heads/master
Commit: 2571cfcf5e057141c033101df039d8a4366b0f33
Parents: 9009802 bef0e9d
Author: Dan Halperin 
Authored: Tue Sep 27 14:57:49 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 27 14:57:49 2016 -0700

--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 120 +++--
 .../beam/sdk/transforms/DoFnTesterTest.java | 456 +++
 2 files changed, 338 insertions(+), 238 deletions(-)
--

Jenkins build is unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1220

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1405

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-679) Bigtable IO integration tests are failing

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-679.

   Resolution: Fixed
 Assignee: Luke Cwik  (was: Jean-Baptiste Onofré)
Fix Version/s: 0.3.0-incubating

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Luke Cwik
>Priority: Critical
> Fix For: 0.3.0-incubating
>
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527323#comment-15527323
 ] 

Luke Cwik commented on BEAM-679:


It turned out that the worker images for Dataflow weren't publicly available 
which caused them to get stuck.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1010: Compress serialized function data.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1010


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Compress serialized function data.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 19e3eff91 -> dc92438fa


Compress serialized function data.

Pickled data is often quite compressible, but this is particularly
useful for concat sources generated for large expansions of filepatterns.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e80dcb11
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e80dcb11
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e80dcb11

Branch: refs/heads/python-sdk
Commit: e80dcb11d5df45e021be4a2c8ff5b84fa1460f91
Parents: 19e3eff
Author: Robert Bradshaw 
Authored: Mon Sep 26 16:24:06 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 16:24:06 2016 -0700

--
 sdks/python/apache_beam/internal/pickler.py | 13 -
 1 file changed, 8 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e80dcb11/sdks/python/apache_beam/internal/pickler.py
--
diff --git a/sdks/python/apache_beam/internal/pickler.py 
b/sdks/python/apache_beam/internal/pickler.py
index 898e04b..30f0b77 100644
--- a/sdks/python/apache_beam/internal/pickler.py
+++ b/sdks/python/apache_beam/internal/pickler.py
@@ -31,6 +31,7 @@ import logging
 import sys
 import traceback
 import types
+import zlib
 
 import dill
 
@@ -182,20 +183,22 @@ logging.getLogger('dill').setLevel(logging.WARN)
 # encoding.  This should be cleaned up.
 def dumps(o):
   try:
-return base64.b64encode(dill.dumps(o))
+s = dill.dumps(o)
   except Exception:  # pylint: disable=broad-except
 dill.dill._trace(True)   # pylint: disable=protected-access
-return base64.b64encode(dill.dumps(o))
+s = dill.dumps(o)
   finally:
 dill.dill._trace(False)  # pylint: disable=protected-access
+  return base64.b64encode(zlib.compress(s))
 
 
-def loads(s):
+def loads(encoded):
+  s = zlib.decompress(base64.b64decode(encoded))
   try:
-return dill.loads(base64.b64decode(s))
+return dill.loads(s)
   except Exception:  # pylint: disable=broad-except
 dill.dill._trace(True)   # pylint: disable=protected-access
-return dill.loads(base64.b64decode(s))
+return dill.loads(s)
   finally:
 dill.dill._trace(False)  # pylint: disable=protected-access
 

[2/2] incubator-beam git commit: Closes #1010

[robertwb]

Closes #1010


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dc92438f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dc92438f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dc92438f

Branch: refs/heads/python-sdk
Commit: dc92438fa038470065994e195d940c0c4c37e13d
Parents: 19e3eff e80dcb1
Author: Robert Bradshaw 
Authored: Tue Sep 27 13:17:19 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Sep 27 13:17:19 2016 -0700

--
 sdks/python/apache_beam/internal/pickler.py | 13 -
 1 file changed, 8 insertions(+), 5 deletions(-)
--

[jira] [Resolved] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-670.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1019: Refactor BundleFactory methods

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1019

Refactor BundleFactory methods

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Remove the inputBundle parameter to createBundle and createKeyedBundle.
This parameter is unless the model is capable of propagating keys
between PTransforms.

Remove the PCollection parameter to createRootBundle. createRootBundle
should be a par

Make [Un]CommmittedBundle#getPCollection nullable. Bundles are utilized
by the runner to control the processing of elements, but may not always
belong to a PCollection. Update ImmutabilityCheckingBundleFactory to
return an underlying bundle as the result of createRootBundle.

Use createBundle in Root transform evaluators.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rm_root_bundle_argument

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1019


commit 7028d9bef52d3a713cb058f6ece8f73530a3717b
Author: Thomas Groh 
Date:   2016-09-27T20:06:35Z

Refactor BundleFactory methods

Remove the inputBundle parameter to createBundle and createKeyedBundle.
This parameter is unless the model is capable of propagating keys
between PTransforms.

Remove the PCollection parameter to createRootBundle. createRootBundle
should be a par

Make [Un]CommmittedBundle#getPCollection nullable. Bundles are utilized
by the runner to control the processing of elements, but may not always
belong to a PCollection. Update ImmutabilityCheckingBundleFactory to
return an underlying bundle as the result of createRootBundle.

Use createBundle in Root transform evaluators.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527281#comment-15527281
 ] 

ASF GitHub Bot commented on BEAM-670:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1018


> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #1018

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1556eb77f -> 900980246


Closes #1018


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90098024
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90098024
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90098024

Branch: refs/heads/master
Commit: 900980246b3ee3bdda8009672d5092a9260f06ba
Parents: 1556eb7 e4b98fd
Author: Dan Halperin 
Authored: Tue Sep 27 13:14:17 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 27 13:14:17 2016 -0700

--
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 23 +++--
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 93 +++-
 2 files changed, 105 insertions(+), 11 deletions(-)
--

[GitHub] incubator-beam pull request #1018: [BEAM-670] BigQueryServicesImpl: fix issu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1018


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: BigQueryServicesImpl: fix issues in insertAll and add better tests

[dhalperi]

BigQueryServicesImpl: fix issues in insertAll and add better tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e4b98fd3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e4b98fd3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e4b98fd3

Branch: refs/heads/master
Commit: e4b98fd39a896a6d3d386d64612f75adab76af8e
Parents: 1556eb7
Author: Dan Halperin 
Authored: Tue Sep 27 10:47:07 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 27 13:14:17 2016 -0700

--
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 23 +++--
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 93 +++-
 2 files changed, 105 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4b98fd3/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
index 7d98401..3862382 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
@@ -543,10 +543,9 @@ class BigQueryServicesImpl implements BigQueryServices {
   backoff);
 }
 
-@Override
-public long insertAll(
-TableReference ref, List rowList, @Nullable List 
insertIdList)
-throws IOException, InterruptedException {
+@VisibleForTesting
+long insertAll(TableReference ref, List rowList, @Nullable 
List insertIdList,
+BackOff backoff, final Sleeper sleeper) throws IOException, 
InterruptedException {
   checkNotNull(ref, "ref");
   if (executor == null) {
 this.executor = options.as(GcsOptions.class).getExecutorService();
@@ -556,8 +555,6 @@ class BigQueryServicesImpl implements BigQueryServices {
 + "as many elements as rowList");
   }
 
-  BackOff backoff = INSERT_BACKOFF_FACTORY.backoff();
-
   long retTotalDataSize = 0;
   List allErrors = new 
ArrayList<>();
   // These lists contain the rows to publish. Initially the contain the 
entire list.
@@ -607,7 +604,7 @@ class BigQueryServicesImpl implements BigQueryServices {
 if (new ApiErrorExtractor().rateLimited(e)) {
   LOG.info("BigQuery insertAll exceeded rate limit, 
retrying");
   try {
-Thread.sleep(backoff.nextBackOffMillis());
+sleeper.sleep(backoff.nextBackOffMillis());
   } catch (InterruptedException interrupted) {
 throw new IOException(
 "Interrupted while waiting before retrying 
insertAll");
@@ -662,16 +659,16 @@ class BigQueryServicesImpl implements BigQueryServices {
   break;
 }
 try {
-  Thread.sleep(backoff.nextBackOffMillis());
+  sleeper.sleep(nextBackoffMillis);
 } catch (InterruptedException e) {
   Thread.currentThread().interrupt();
   throw new IOException(
   "Interrupted while waiting before retrying insert of " + 
retryRows);
 }
-LOG.info("Retrying failed inserts to BigQuery");
 rowsToPublish = retryRows;
 idsToPublish = retryIds;
 allErrors.clear();
+LOG.info("Retrying {} failed inserts to BigQuery", 
rowsToPublish.size());
   }
   if (!allErrors.isEmpty()) {
 throw new IOException("Insert failed: " + allErrors);
@@ -679,6 +676,14 @@ class BigQueryServicesImpl implements BigQueryServices {
 return retTotalDataSize;
   }
 }
+
+@Override
+public long insertAll(
+TableReference ref, List rowList, @Nullable List 
insertIdList)
+throws IOException, InterruptedException {
+  return insertAll(
+  ref, rowList, insertIdList, INSERT_BACKOFF_FACTORY.backoff(), 
Sleeper.DEFAULT);
+}
   }
 
   private static class BigQueryJsonReaderImpl implements BigQueryJsonReader {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4b98fd3/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
--
diff --git 

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527094#comment-15527094
 ] 

Jean-Baptiste Onofré commented on BEAM-679:
---

Jenkins looks stuck on Bigtable ITs.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526915#comment-15526915
 ] 

ASF GitHub Bot commented on BEAM-670:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1018

[BEAM-670] BigQueryServicesImpl: fix issues in insertAll and add better 
tests

R: @lukecwik
CC: @vikkyrk 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam bigquery-tableinserter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1018.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1018


commit 3581a4e1a67fcb37cdfe5fec8e7973fbad3617d5
Author: Dan Halperin 
Date:   2016-09-27T17:47:07Z

BigQueryServicesImpl: fix issues in insertAll and add better tests




> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1018: [BEAM-670] BigQueryServicesImpl: fix issu...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1018

[BEAM-670] BigQueryServicesImpl: fix issues in insertAll and add better 
tests

R: @lukecwik
CC: @vikkyrk 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam bigquery-tableinserter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1018.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1018


commit 3581a4e1a67fcb37cdfe5fec8e7973fbad3617d5
Author: Dan Halperin 
Date:   2016-09-27T17:47:07Z

BigQueryServicesImpl: fix issues in insertAll and add better tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #991: Allow .whl files to be staged with --extra...

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/991


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526835#comment-15526835
 ] 

Jean-Baptiste Onofré commented on BEAM-679:
---

No problem, thanks for the fix Luke. I'm testing on Jenkins right now, I will 
resolve this Jira if it passes.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Update Dataflow worker image for Apache Beam

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 68010aba1 -> 1556eb77f


Update Dataflow worker image for Apache Beam


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a11c9618
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a11c9618
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a11c9618

Branch: refs/heads/master
Commit: a11c9618996b370c42d015c6d238532c5393aacb
Parents: 68010ab
Author: Luke Cwik 
Authored: Mon Sep 26 19:25:39 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 10:11:44 2016 -0700

--
 pom.xml  | 2 +-
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a11c9618/pom.xml
--
diff --git a/pom.xml b/pom.xml
index cc93bb9..8104e11 100644
--- a/pom.xml
+++ b/pom.xml
@@ -122,7 +122,7 @@
 2.4
 4.11
 1.9.5
-4.1.1.Final
+4.1.3.Final
 1.4.0.Final
 3.0.0
 v1-rev10-1.22.0

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a11c9618/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index d067b61..54c95a7 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -206,9 +206,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160826";
+  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160926";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160826";
+  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160926";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526785#comment-15526785
 ] 

ASF GitHub Bot commented on BEAM-679:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1012


> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526797#comment-15526797
 ] 

Jean-Baptiste Onofré commented on BEAM-679:
---

Let me check on Jenkins.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526796#comment-15526796
 ] 

Jean-Baptiste Onofré commented on BEAM-679:
---

Great, thanks !

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526790#comment-15526790
 ] 

Luke Cwik commented on BEAM-679:


Got permissions, BigtableWriteIT passed. Next postcommit should validate these 
findings.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1012: [BEAM-679] Update Dataflow worker image f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1012


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-679] Update Dataflow worker image for Apache Beam / Fix Netty Version

[lcwik]

[BEAM-679] Update Dataflow worker image for Apache Beam / Fix Netty Version

This closes #1012


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1556eb77
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1556eb77
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1556eb77

Branch: refs/heads/master
Commit: 1556eb77fb8e902ac3af6514f61457c83fe5d2d0
Parents: 68010ab a11c961
Author: Luke Cwik 
Authored: Tue Sep 27 10:12:30 2016 -0700
Committer: Luke Cwik 
Committed: Tue Sep 27 10:12:30 2016 -0700

--
 pom.xml  | 2 +-
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--

[jira] [Commented] (BEAM-683) Make BZIP compressed files splittable

[Tim Sears (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526782#comment-15526782
 ] 

Tim Sears commented on BEAM-683:


I'm not sure, could you point me to the module where that happens?

> Make BZIP compressed files splittable 
> --
>
> Key: BEAM-683
> URL: https://issues.apache.org/jira/browse/BEAM-683
> Project: Beam
>  Issue Type: Wish
>Reporter: Tim Sears
>Priority: Minor
>   Original Estimate: 10h
>  Remaining Estimate: 10h
>
> Bzip2 is compressed as blocks, so it should be possible to do dynamic 
> splitting. To do this: Seek to a location in the bzip, then keep seeking 
> until you find the 6 byte block-start sequence 0x314159265359 (which is the 
> 12 digit approximation of pi). You can use a bzip2 decompressor from that 
> point onwards.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-679) Bigtable IO integration tests are failing

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526647#comment-15526647
 ] 

Luke Cwik commented on BEAM-679:


Trying to rerun the integration test locally to verify.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-685) TextIO path pattern error on Windows OS

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-685:
-
Description: 
It seems that the pattern {{some-output-file-*}} is rejected on Windows, or at 
least by nio's Windows path code. This could particularly affect Windows users 
getting started with the DirectRunner.

Apologies for lack of steps to certainly repro - this is a quick write up of an 
issue a user had in person.

  was:
It seems that the pattern {{some-output-file-*}} is rejected on Windows, or at 
least by nio's Windows path code. This could particularly affect Windows users 
getting started.

Apologies for lack of steps to certainly repro - this is a quick write up of an 
issue a user had in person.


> TextIO path pattern error on Windows OS
> ---
>
> Key: BEAM-685
> URL: https://issues.apache.org/jira/browse/BEAM-685
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> It seems that the pattern {{some-output-file-*}} is rejected on Windows, or 
> at least by nio's Windows path code. This could particularly affect Windows 
> users getting started with the DirectRunner.
> Apologies for lack of steps to certainly repro - this is a quick write up of 
> an issue a user had in person.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-685) TextIO path pattern error on Windows OS

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-685:


 Summary: TextIO path pattern error on Windows OS
 Key: BEAM-685
 URL: https://issues.apache.org/jira/browse/BEAM-685
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles


It seems that the pattern {{some-output-file-*}} is rejected on Windows, or at 
least by nio's Windows path code. This could particularly affect Windows users 
getting started.

Apologies for lack of steps to certainly repro - this is a quick write up of an 
issue a user had in person.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-475) High-quality javadoc for Beam

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526249#comment-15526249
 ] 

Kenneth Knowles commented on BEAM-475:
--

See also BEAM-385 for raising the visibility.

> High-quality javadoc for Beam
> -
>
> Key: BEAM-475
> URL: https://issues.apache.org/jira/browse/BEAM-475
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> We should have good Javadoc for Beam!
> Current snapshot: http://beam.incubator.apache.org/javadoc/0.1.0-incubating/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-684) De-Dataflow Beam

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-684:
-
Labels: easy easyfix starter  (was: )

> De-Dataflow Beam
> 
>
> Key: BEAM-684
> URL: https://issues.apache.org/jira/browse/BEAM-684
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>  Labels: easy, easyfix, starter
>
> {code}
> grep -i dataflow sdks/java/core/**/*.java | wc -l
>  136
> {code}
> Most of those uses of the word {{dataflow}} are wrong. We should really 
> finish converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526111#comment-15526111
 ] 

Jean-Baptiste Onofré edited comment on BEAM-679 at 9/27/16 1:23 PM:


It seems the problem is caused by these commits: 

https://github.com/apache/incubator-beam/commit/a447d130172ec7c270fc9f1d16a2f7404898461e

https://github.com/apache/incubator-beam/commit/f93ca9ce803a8847a7178ff0d7c5e1631bed8f2d

I'm testing locally to re-introduce the dependencies to see if it helps.


was (Author: jbonofre):
It seems the problem is caused by this commit: 
https://github.com/apache/incubator-beam/commit/a447d130172ec7c270fc9f1d16a2f7404898461e

I'm testing locally to re-introduce the dependencies to see if it helps.

> Bigtable IO integration tests are failing
> -
>
> Key: BEAM-679
> URL: https://issues.apache.org/jira/browse/BEAM-679
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>
> Bigtable ITests are failing with the following issue:
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class 
> com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
> {code}
> I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1217

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-683) Make BZIP compressed files splittable

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526052#comment-15526052
 ] 

Jean-Baptiste Onofré commented on BEAM-683:
---

Kind of similar with what we did on Zip files ?

> Make BZIP compressed files splittable 
> --
>
> Key: BEAM-683
> URL: https://issues.apache.org/jira/browse/BEAM-683
> Project: Beam
>  Issue Type: Wish
>Reporter: Tim Sears
>Priority: Minor
>   Original Estimate: 10h
>  Remaining Estimate: 10h
>
> Bzip2 is compressed as blocks, so it should be possible to do dynamic 
> splitting. To do this: Seek to a location in the bzip, then keep seeking 
> until you find the 6 byte block-start sequence 0x314159265359 (which is the 
> 12 digit approximation of pi). You can use a bzip2 decompressor from that 
> point onwards.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1402

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1402

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1017: Beam 517 - Check versions of pip and cyth...

[mbuccini]

GitHub user mbuccini opened a pull request:

https://github.com/apache/incubator-beam/pull/1017

Beam 517 - Check versions of pip and cython

I have added a check so that the setup fails if pip <= 0.7.0 is not 
installed. 

In addition, I have added cython as an optional dependency, so that it's 
installed only if the user decides to do so, e.g., via 

`pip install -e .[cython]`

The Readme file has also been updated, although I think that it may need 
some refactoring in the future, when/if the project is packaged and deployed on 
pypi as apache-beam. In that case, it would be possible to execute something 
like:

`pip install apache-beam[cython]`

---
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

For more details about the issue, see the jira issue: 
https://issues.apache.org/jira/browse/BEAM-517

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mbuccini/incubator-beam beam-517

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1017


commit 588b4a35c6d20ade7965c187887c83515eef16c1
Author: mbuccini 
Date:   2016-09-27T08:30:38Z

Fixed pip requirement and set cython as extra package.

commit 66caeb8c6276e2b349ed68f161e85dc910d515a9
Author: mbuccini 
Date:   2016-09-27T08:50:16Z

fixed README for cython/pip versions

commit cb4efec4cc1c400b24e062b007374c0dd7b33aa6
Author: mbuccini 
Date:   2016-09-27T08:54:56Z

Fixed anchors/titles

commit c168e69491d0c1df8aeed0574dee75e375dea27f
Author: mbuccini 
Date:   2016-09-27T08:57:14Z

Fixed typo

commit 8adbd8f4ea18b0d91a5c55914fed31d5b3f70c12
Author: mbuccini 
Date:   2016-09-27T09:34:05Z

Removed section about clone/install Apache Beam




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-517) Check versions of pip and cython

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525627#comment-15525627
 ] 

ASF GitHub Bot commented on BEAM-517:
-

GitHub user mbuccini opened a pull request:

https://github.com/apache/incubator-beam/pull/1017

Beam 517 - Check versions of pip and cython

I have added a check so that the setup fails if pip <= 0.7.0 is not 
installed. 

In addition, I have added cython as an optional dependency, so that it's 
installed only if the user decides to do so, e.g., via 

`pip install -e .[cython]`

The Readme file has also been updated, although I think that it may need 
some refactoring in the future, when/if the project is packaged and deployed on 
pypi as apache-beam. In that case, it would be possible to execute something 
like:

`pip install apache-beam[cython]`

---
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

For more details about the issue, see the jira issue: 
https://issues.apache.org/jira/browse/BEAM-517

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mbuccini/incubator-beam beam-517

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1017


commit 588b4a35c6d20ade7965c187887c83515eef16c1
Author: mbuccini 
Date:   2016-09-27T08:30:38Z

Fixed pip requirement and set cython as extra package.

commit 66caeb8c6276e2b349ed68f161e85dc910d515a9
Author: mbuccini 
Date:   2016-09-27T08:50:16Z

fixed README for cython/pip versions

commit cb4efec4cc1c400b24e062b007374c0dd7b33aa6
Author: mbuccini 
Date:   2016-09-27T08:54:56Z

Fixed anchors/titles

commit c168e69491d0c1df8aeed0574dee75e375dea27f
Author: mbuccini 
Date:   2016-09-27T08:57:14Z

Fixed typo

commit 8adbd8f4ea18b0d91a5c55914fed31d5b3f70c12
Author: mbuccini 
Date:   2016-09-27T09:34:05Z

Removed section about clone/install Apache Beam




> Check versions of pip and cython
> 
>
> Key: BEAM-517
> URL: https://issues.apache.org/jira/browse/BEAM-517
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Python SDK depends on pip and cython however it does not check the versions 
> of these.
> Some of the pip flags does not exist in older versions:
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/28#issuecomment-236382953
> (Note: Even though the above issue was reported by the user in a different 
> repo it is related to the apache beam sdk)
> Similarly with cython, SDK supports running with or without Cython. Because 
> of that reason it is not list it as a requirement in the setup.py file. 
> However, with an old version of cython SDK might fail.
> To avoid the above problem: In the SDK check the version of these packages 
> and show a warning to upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1216

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1401

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Kinesis #1401

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_MavenVerify #1401

[Apache Jenkins Server]

See