[GitHub] beam pull request #3293: [BEAM-2406] Fix NullPointerException when writing a...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/3293

[BEAM-2406] Fix NullPointerException when writing an empty table

When writing to an empty table, BigQueryIO was throwing a NPE because 
WritePartitions was returning null. This was a regression in previous behavior.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam fix_bq_npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3293.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3293


commit e7dc19e118f00e3706c15e19ece7e0f34de1d72e
Author: Reuven Lax 
Date:   2017-06-03T01:29:59Z

Fix crash.

commit 2c6e4b443110b0fcc1334e9d1fac7493b15a4982
Author: Reuven Lax 
Date:   2017-06-05T05:11:41Z

Fix test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/5] beam git commit: Add translation convenience for non-composites

[kenn]

Add translation convenience for non-composites


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9c3fbc37
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9c3fbc37
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9c3fbc37

Branch: refs/heads/master
Commit: 9c3fbc37e9b703c381d284808dfcfef67bb8b531
Parents: f1386c1
Author: Kenneth Knowles 
Authored: Thu Jun 1 13:01:20 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 10:06:52 2017 -0700

--
 .../core/construction/PTransformTranslation.java| 12 
 1 file changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9c3fbc37/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 99d1e85..b2f06ac 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -24,6 +24,7 @@ import com.google.common.collect.ImmutableMap;
 import com.google.protobuf.Any;
 import com.google.protobuf.Message;
 import java.io.IOException;
+import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 import java.util.ServiceLoader;
@@ -120,6 +121,17 @@ public class PTransformTranslation {
 return transformBuilder.build();
   }
 
+  /**
+   * Translates a non-composite {@link AppliedPTransform} into a runner API 
proto.
+   *
+   * Does not register the {@code appliedPTransform} within the provided 
{@link SdkComponents}.
+   */
+  static RunnerApi.PTransform toProto(
+  AppliedPTransform appliedPTransform, SdkComponents components) 
throws IOException {
+return toProto(
+appliedPTransform, Collections.emptyList(), components);
+  }
+
   private static String toProto(TupleTag tag) {
 return tag.getId();
   }

[5/5] beam git commit: This closes #3283: [BEAM-3271] Add TestStream translation

[kenn]

This closes #3283: [BEAM-3271] Add TestStream translation

  Add TestStream translation
  Expose internal constructors for TestStream events
  Add TestStreamPayload to Runner API proto
  Add translation convenience for non-composites


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0cba43ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0cba43ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0cba43ee

Branch: refs/heads/master
Commit: 0cba43ee26c00ae89f8e681f7fb1f0294aeadb0e
Parents: 462335c 0cedc61
Author: Kenneth Knowles 
Authored: Sun Jun 4 21:30:10 2017 -0700
Committer: Kenneth Knowles 
Committed: Sun Jun 4 21:30:10 2017 -0700

--
 .../construction/PTransformTranslation.java |  13 ++
 .../construction/TestStreamTranslation.java | 156 +++
 .../construction/TestStreamTranslationTest.java | 129 +++
 .../src/main/proto/beam_runner_api.proto|  33 
 .../org/apache/beam/sdk/testing/TestStream.java |  60 +--
 5 files changed, 375 insertions(+), 16 deletions(-)
--

[GitHub] beam pull request #3283: [BEAM-3271] Add TestStream translation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3283


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/5] beam git commit: Expose internal constructors for TestStream events

[kenn]

Expose internal constructors for TestStream events

These are needed for deserialization from proto.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/47cea784
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/47cea784
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/47cea784

Branch: refs/heads/master
Commit: 47cea78496a9a464d8cea7943a2f741c03692612
Parents: f5e30c5
Author: Kenneth Knowles 
Authored: Thu Jun 1 19:17:58 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 10:06:52 2017 -0700

--
 .../org/apache/beam/sdk/testing/TestStream.java | 60 ++--
 1 file changed, 44 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/47cea784/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
index d41b9ef..9ad8fd8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
@@ -24,8 +24,10 @@ import static 
com.google.common.base.Preconditions.checkNotNull;
 import com.google.auto.value.AutoValue;
 import com.google.common.collect.ImmutableList;
 import java.util.List;
+import java.util.Objects;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -86,8 +88,8 @@ public final class TestStream extends PTransform {
 /**
  * Adds the specified elements to the source with timestamp equal to the 
current watermark.
  *
- * @return A {@link TestStream.Builder} like this one that will add the 
provided elements
- * after all earlier events have completed.
+ * @return A {@link TestStream.Builder} like this one that will add the 
provided elements after
+ * all earlier events have completed.
  */
 @SafeVarargs
 public final Builder addElements(T element, T... elements) {
@@ -103,8 +105,8 @@ public final class TestStream extends PTransform {
 /**
  * Adds the specified elements to the source with the provided timestamps.
  *
- * @return A {@link TestStream.Builder} like this one that will add the 
provided elements
- * after all earlier events have completed.
+ * @return A {@link TestStream.Builder} like this one that will add the 
provided elements after
+ * all earlier events have completed.
  */
 @SafeVarargs
 public final Builder addElements(
@@ -136,7 +138,7 @@ public final class TestStream extends PTransform {
  * BoundedWindow#TIMESTAMP_MAX_VALUE} or beyond.
  *
  * @return A {@link TestStream.Builder} like this one that will advance 
the watermark to the
- * specified point after all earlier events have completed.
+ * specified point after all earlier events have completed.
  */
 public Builder advanceWatermarkTo(Instant newWatermark) {
   checkArgument(
@@ -146,10 +148,11 @@ public final class TestStream extends 
PTransform {
   "The Watermark cannot progress beyond the maximum. Got: %s. Maximum: 
%s",
   newWatermark,
   BoundedWindow.TIMESTAMP_MAX_VALUE);
-  ImmutableList newEvents = ImmutableList.builder()
-  .addAll(events)
-  .add(WatermarkEvent.advanceTo(newWatermark))
-  .build();
+  ImmutableList newEvents =
+  ImmutableList.builder()
+  .addAll(events)
+  .add(WatermarkEvent.advanceTo(newWatermark))
+  .build();
   return new Builder(coder, newEvents, newWatermark);
 }
 
@@ -157,7 +160,7 @@ public final class TestStream extends PTransform {
  * Advance the processing time by the specified amount.
  *
  * @return A {@link TestStream.Builder} like this one that will advance 
the processing time by
- * the specified amount after all earlier events have completed.
+ * the specified amount after all earlier events have completed.
  */
 public Builder advanceProcessingTime(Duration amount) {
   checkArgument(
@@ -194,9 +197,7 @@ public final class TestStream extends PTransform {
 EventType getType();
   }
 
-  /**
-   * The types of {@link Event} that are supported by 

[1/5] beam git commit: Add TestStream translation

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 462335caf -> 0cba43ee2


Add TestStream translation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0cedc61f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0cedc61f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0cedc61f

Branch: refs/heads/master
Commit: 0cedc61ffb59a08a4b5205a5a224fd9fa906f7a7
Parents: 47cea78
Author: Kenneth Knowles 
Authored: Tue May 30 14:41:46 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 10:06:52 2017 -0700

--
 .../construction/PTransformTranslation.java |   1 +
 .../construction/TestStreamTranslation.java | 156 +++
 .../construction/TestStreamTranslationTest.java | 129 +++
 3 files changed, 286 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0cedc61f/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index b2f06ac..fd3f9f3 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -50,6 +50,7 @@ public class PTransformTranslation {
   public static final String GROUP_BY_KEY_TRANSFORM_URN = 
"urn:beam:transform:groupbykey:v1";
   public static final String READ_TRANSFORM_URN = "urn:beam:transform:read:v1";
   public static final String WINDOW_TRANSFORM_URN = 
"urn:beam:transform:window:v1";
+  public static final String TEST_STREAM_TRANSFORM_URN = 
"urn:beam:transform:teststream:v1";
 
   // Less well-known. And where shall these live?
   public static final String WRITE_FILES_TRANSFORM_URN = 
"urn:beam:transform:write_files:0.1";

http://git-wip-us.apache.org/repos/asf/beam/blob/0cedc61f/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
new file mode 100644
index 000..90e6304
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * Utility methods for translating a {@link TestStream} to and from {@link 
RunnerApi}
+ * representations.
+ */
+public class TestStreamTranslation {
+
+  static  RunnerApi.TestStreamPayload testStreamToPayload(
+  TestStream transform, SdkComponents components) throws IOException {
+String coderId = 

[4/5] beam git commit: Add TestStreamPayload to Runner API proto

[kenn]

Add TestStreamPayload to Runner API proto


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f5e30c52
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f5e30c52
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f5e30c52

Branch: refs/heads/master
Commit: f5e30c52b46787133651463f343c02113f08c63c
Parents: 9c3fbc3
Author: Kenneth Knowles 
Authored: Thu Jun 1 16:32:47 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 10:06:52 2017 -0700

--
 .../src/main/proto/beam_runner_api.proto| 33 
 1 file changed, 33 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f5e30c52/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index acc4a35..87e33f0 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -325,6 +325,39 @@ message CombinePayload {
   map side_inputs = 4;
 }
 
+// The payload for the test-only primitive TestStream
+message TestStreamPayload {
+
+  // (Required) the coder for elements in the TestStream events
+  string coder_id = 1;
+
+  repeated Event events = 2;
+
+  message Event {
+oneof event {
+  AdvanceWatermark watermark_event = 1;
+  AdvanceProcessingTime processing_time_event = 2;
+  AddElements element_event = 3;
+}
+
+message AdvanceWatermark {
+  int64 new_watermark = 1;
+}
+
+message AdvanceProcessingTime {
+  int64 advance_duration = 1;
+}
+
+message AddElements {
+  repeated TimestampedElement elements = 1;
+}
+  }
+
+  message TimestampedElement {
+bytes encoded_element = 1;
+int64 timestamp = 2;
+  }
+}
 // The payload for the special-but-not-primitive WriteFiles transform.
 message WriteFilesPayload {
 

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036469#comment-16036469
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3292

[BEAM-79] Merge master into gearpump-runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3292


commit 66e564e507b1ad5b9e379e043a193b556e1f3e76
Author: Borisa Zivkovic 
Date:   2017-05-15T08:24:10Z

[BEAM-1702] Document how to use BigtableIO with local emulator

commit 8ef812defc5237934f4f3efa6bd8c4c4240cd7d0
Author: Dan Halperin 
Date:   2017-05-18T20:07:13Z

This closes #3141

commit a8d2125e5783a556056e88dad8fe3c0a397920d5
Author: huafengw 
Date:   2017-05-09T03:21:44Z

[BEAM-659] WindowFn#isCompatible should provide a meaningful reason

commit 8c572ef0b998a4fa529e2f4ba0e622b3db9f4ec6
Author: Kenneth Knowles 
Date:   2017-05-18T20:27:59Z

This closes #2985: WindowFn#isCompatible should provide a meaningful reason

commit 26a615e885b39f92dc81d383c8115906c280187a
Author: Andrew Martin 
Date:   2017-05-18T19:52:09Z

Remove incorrect `Message` bound on writeAvros

commit 6a7eeeb932ffb33c876c0d11d7f325c47606d8de
Author: Eugene Kirpichov 
Date:   2017-05-18T20:52:33Z

This closes #3176: Remove incorrect `Message` bound on writeAvros

commit 790e7fe6653b926044d3dfecdccbc2fda9c998f0
Author: Thomas Groh 
Date:   2017-03-21T22:06:58Z

Add ParDos

Add ParDoPayloadTranslator to PTransformTranslator

commit 2e0cf008303ff5ed48268550b9fe3cd1aa748c15
Author: Thomas Groh 
Date:   2017-05-18T22:41:17Z

This closes #2597

commit 7a2e829c849d3828353994b8eddca4784a900aca
Author: Eugene Kirpichov 
Date:   2017-05-15T22:54:03Z

Splits SplittableParDo into a core-construction part and a KWI-related part

commit c93b3aa3d2081558a776fa12c564efb5f15090ab
Author: Eugene Kirpichov 
Date:   2017-05-18T23:15:00Z

This closes #3156

commit 54808be14ff2e86088807adc3c0dc0965c8a6f03
Author: Dan Halperin 
Date:   2017-05-18T22:43:31Z

[BEAM-2318] Ensure that tables that don't exist are not created

We use the same test table name in two tests: one that expects the table 
not to exist, another that
creates the table. Obviously, the correctness of this will depend on the 
order in which unit tests
are executed.

Fix the flake by using different and more explicit test names.

commit f458065dad0429ba2639510ea460cf7f2679ed36
Author: Dan Halperin 
Date:   2017-05-18T23:41:41Z

This closes #3179

commit 1da8da79dcc623c53b35d97419566b736447f6b6
Author: Dan Halperin 
Date:   2017-05-18T20:19:29Z

[BEAM-2391] Clone Scan in HBaseReader

commit 49245080ad6393b5678077caa047f7a6ef0efffc
Author: Ismaël Mejía 
Date:   2017-05-19T08:50:06Z

This closes #3178

commit f969ea443c90345cf8c33f953c399e46223e8d03
Author: Luke Cwik 
Date:   2017-05-15T20:39:12Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

commit ea5b9dabbfb55ff47a4f5983a8cbf04b0949c77c
Author: Luke Cwik 
Date:   2017-05-19T17:37:10Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

This closes #3152

commit 3efb34b8322ac5933d9d71255d67e99d8e8361d7
Author: Ahmet Altay 
Date:   2017-05-19T02:00:00Z

update dataflow python container

commit a58275a2f904ab1304de7ca5295f27a4f6df6f1d
Author: Ahmet Altay 
Date:   2017-05-19T17:43:15Z

This closes #3182

commit bac7bde5f878e6b2104c2e7cf3d01894a8f1b71d
Author: Mark Liu 
Date:   2017-05-17T21:24:25Z

[BEAM-2164] Fix 

[GitHub] beam pull request #3292: [BEAM-79] Merge master into gearpump-runner

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3292

[BEAM-79] Merge master into gearpump-runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3292


commit 66e564e507b1ad5b9e379e043a193b556e1f3e76
Author: Borisa Zivkovic 
Date:   2017-05-15T08:24:10Z

[BEAM-1702] Document how to use BigtableIO with local emulator

commit 8ef812defc5237934f4f3efa6bd8c4c4240cd7d0
Author: Dan Halperin 
Date:   2017-05-18T20:07:13Z

This closes #3141

commit a8d2125e5783a556056e88dad8fe3c0a397920d5
Author: huafengw 
Date:   2017-05-09T03:21:44Z

[BEAM-659] WindowFn#isCompatible should provide a meaningful reason

commit 8c572ef0b998a4fa529e2f4ba0e622b3db9f4ec6
Author: Kenneth Knowles 
Date:   2017-05-18T20:27:59Z

This closes #2985: WindowFn#isCompatible should provide a meaningful reason

commit 26a615e885b39f92dc81d383c8115906c280187a
Author: Andrew Martin 
Date:   2017-05-18T19:52:09Z

Remove incorrect `Message` bound on writeAvros

commit 6a7eeeb932ffb33c876c0d11d7f325c47606d8de
Author: Eugene Kirpichov 
Date:   2017-05-18T20:52:33Z

This closes #3176: Remove incorrect `Message` bound on writeAvros

commit 790e7fe6653b926044d3dfecdccbc2fda9c998f0
Author: Thomas Groh 
Date:   2017-03-21T22:06:58Z

Add ParDos

Add ParDoPayloadTranslator to PTransformTranslator

commit 2e0cf008303ff5ed48268550b9fe3cd1aa748c15
Author: Thomas Groh 
Date:   2017-05-18T22:41:17Z

This closes #2597

commit 7a2e829c849d3828353994b8eddca4784a900aca
Author: Eugene Kirpichov 
Date:   2017-05-15T22:54:03Z

Splits SplittableParDo into a core-construction part and a KWI-related part

commit c93b3aa3d2081558a776fa12c564efb5f15090ab
Author: Eugene Kirpichov 
Date:   2017-05-18T23:15:00Z

This closes #3156

commit 54808be14ff2e86088807adc3c0dc0965c8a6f03
Author: Dan Halperin 
Date:   2017-05-18T22:43:31Z

[BEAM-2318] Ensure that tables that don't exist are not created

We use the same test table name in two tests: one that expects the table 
not to exist, another that
creates the table. Obviously, the correctness of this will depend on the 
order in which unit tests
are executed.

Fix the flake by using different and more explicit test names.

commit f458065dad0429ba2639510ea460cf7f2679ed36
Author: Dan Halperin 
Date:   2017-05-18T23:41:41Z

This closes #3179

commit 1da8da79dcc623c53b35d97419566b736447f6b6
Author: Dan Halperin 
Date:   2017-05-18T20:19:29Z

[BEAM-2391] Clone Scan in HBaseReader

commit 49245080ad6393b5678077caa047f7a6ef0efffc
Author: Ismaël Mejía 
Date:   2017-05-19T08:50:06Z

This closes #3178

commit f969ea443c90345cf8c33f953c399e46223e8d03
Author: Luke Cwik 
Date:   2017-05-15T20:39:12Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

commit ea5b9dabbfb55ff47a4f5983a8cbf04b0949c77c
Author: Luke Cwik 
Date:   2017-05-19T17:37:10Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

This closes #3152

commit 3efb34b8322ac5933d9d71255d67e99d8e8361d7
Author: Ahmet Altay 
Date:   2017-05-19T02:00:00Z

update dataflow python container

commit a58275a2f904ab1304de7ca5295f27a4f6df6f1d
Author: Ahmet Altay 
Date:   2017-05-19T17:43:15Z

This closes #3182

commit bac7bde5f878e6b2104c2e7cf3d01894a8f1b71d
Author: Mark Liu 
Date:   2017-05-17T21:24:25Z

[BEAM-2164] Fix generate-sources.sh if directory contains space

commit ce00d2466a9495f4eb262b5f7d2afec5788fcfe2
Author: Kenneth Knowles 
Date:   2017-05-19T17:44:54Z

This closes #3171: [BEAM-2164] Fix generate-sources.sh if 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4010

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #44

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #94

[Apache Jenkins Server]

See 


--
[...truncated 2.54 MB...]
2017-06-05T00:35:22.387 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/groovy-maven-plugin/2.0/groovy-maven-plugin-2.0.jar
 (48 KB at 1478.3 KB/sec)
2017-06-05T00:35:22.387 [INFO] 
2017-06-05T00:35:22.387 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-05T00:35:22.402 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-05T00:35:22.433 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 46.7 KB/sec)
2017-06-05T00:35:22.433 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-05T00:35:22.465 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 351.7 KB/sec)
2017-06-05T00:35:22.465 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-05T00:35:22.496 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 414.0 KB/sec)
2017-06-05T00:35:22.496 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-05T00:35:22.527 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 70.1 KB/sec)
2017-06-05T00:35:22.527 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-05T00:35:22.558 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 50.7 KB/sec)
2017-06-05T00:35:22.558 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-05T00:35:22.590 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 164.1 KB/sec)
2017-06-05T00:35:22.590 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-05T00:35:22.621 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 80.5 KB/sec)
2017-06-05T00:35:22.621 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-05T00:35:22.652 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 110.5 KB/sec)
2017-06-05T00:35:22.652 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-05T00:35:22.683 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 32.4 KB/sec)
2017-06-05T00:35:22.683 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-05T00:35:22.715 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 53.0 KB/sec)
2017-06-05T00:35:22.715 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-05T00:35:22.746 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 56.9 KB/sec)
2017-06-05T00:35:22.746 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-05T00:35:22.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 85.6 KB/sec)
2017-06-05T00:35:22.777 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-05T00:35:22.808 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 568.3 KB/sec)
2017-06-05T00:35:22.808 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-05T00:35:22.840 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 294.6 KB/sec)
2017-06-05T00:35:22.840 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-05T00:35:22.871 [INFO] Downloaded: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #44

[Apache Jenkins Server]

See 


--
[...truncated 1.41 MB...]
  [javadoc]   location: package java.util.concurrent
  [javadoc] 
:29:
 error: package java.util.function does not exist
  [javadoc] import java.util.function.Supplier;
  [javadoc]  ^
  [javadoc] 
:56:
 error: cannot find symbol
  [javadoc]   private final Supplier processBundleInstructionIdSupplier;
  [javadoc] ^
  [javadoc]   symbol:   class Supplier
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:61:
 error: cannot find symbol
  [javadoc]   private CompletableFuture readFuture;
  [javadoc]   ^
  [javadoc]   symbol:   class CompletableFuture
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:65:
 error: cannot find symbol
  [javadoc]   Supplier processBundleInstructionIdSupplier,
  [javadoc]   ^
  [javadoc]   symbol:   class Supplier
  [javadoc]   location: class BeamFnDataReadRunner
  [javadoc]   where OutputT is a type-variable:
  [javadoc] OutputT extends Object declared in class BeamFnDataReadRunner
  [javadoc] 
:51:
 warning - Tag @link: reference not found: DoFn.FinishBundle @FinishBundle
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:33:
 warning - Tag @link: reference not found: DoFn.StartBundle @StartBundle
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessElement @ProcessElement
  [javadoc] 
:39:
 warning - Tag @link: reference not found: DoFn.ProcessContext
  [javadoc] 
:45:
 warning - Tag @link: reference not found: DoFn.OnTimer @OnTimer
  [javadoc] 
:51:
 warning - Tag @link: reference 

[GitHub] beam pull request #3291: Make all Hadoop related IOs use the same version pr...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3291

Make all Hadoop related IOs use the same version property

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam align-hadoop-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3291


commit affd54d582d0dcbd8dd04cbcb40a3c4bf745dce3
Author: Ismaël Mejía 
Date:   2017-06-04T20:55:05Z

Make all Hadoop related IOs use the same version property




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4009

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #93

[Apache Jenkins Server]

See 


--
[...truncated 2.53 MB...]
2017-06-04T18:35:00.992 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/groovy-maven-plugin/2.0/groovy-maven-plugin-2.0.jar
 (48 KB at 2628.1 KB/sec)
2017-06-04T18:35:00.998 [INFO] 
2017-06-04T18:35:00.998 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-04T18:35:01.003 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-04T18:35:01.016 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 111.3 KB/sec)
2017-06-04T18:35:01.020 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-04T18:35:01.032 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 938.0 KB/sec)
2017-06-04T18:35:01.036 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-04T18:35:01.044 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1604.2 KB/sec)
2017-06-04T18:35:01.049 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-04T18:35:01.060 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 197.5 KB/sec)
2017-06-04T18:35:01.065 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-04T18:35:01.071 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 262.2 KB/sec)
2017-06-04T18:35:01.076 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-04T18:35:01.087 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 477.3 KB/sec)
2017-06-04T18:35:01.090 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-04T18:35:01.100 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 249.5 KB/sec)
2017-06-04T18:35:01.104 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-04T18:35:01.114 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-04T18:35:01.119 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-04T18:35:01.136 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 59.0 KB/sec)
2017-06-04T18:35:01.141 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-04T18:35:01.151 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 169.6 KB/sec)
2017-06-04T18:35:01.156 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-04T18:35:01.166 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 176.4 KB/sec)
2017-06-04T18:35:01.170 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-04T18:35:01.180 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 265.4 KB/sec)
2017-06-04T18:35:01.186 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-04T18:35:01.197 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1601.5 KB/sec)
2017-06-04T18:35:01.201 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-04T18:35:01.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 1047.5 KB/sec)
2017-06-04T18:35:01.214 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-04T18:35:01.224 [INFO] Downloaded: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #43

[Apache Jenkins Server]

See 


--
[...truncated 899.35 KB...]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(f578cad3e69fce00): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #43

[Apache Jenkins Server]

See 


--
[...truncated 899.57 KB...]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(3f58546e00dc6367): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.createSession(GrpcSpannerRpc.java:322)
at 

[jira] [Commented] (BEAM-1498) Use Flink-native side outputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036286#comment-16036286
 ] 

ASF GitHub Bot commented on BEAM-1498:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/3290

[BEAM-1498] Use Flink-native side outputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1498

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3290


commit 0a8ad92b9dc5bc90572702592e1b203f4dd909f7
Author: JingsongLi 
Date:   2017-06-04T13:56:10Z

[BEAM-1498] Use Flink-native side outputs




> Use Flink-native side outputs
> -
>
> Key: BEAM-1498
> URL: https://issues.apache.org/jira/browse/BEAM-1498
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> Once Flink has support for side outputs we should use them instead of 
> manually dealing with the {{RawUnionValues}}.
> Side outputs for Flink is being tracked in 
> https://issues.apache.org/jira/browse/FLINK-4460.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3290: [BEAM-1498] Use Flink-native side outputs

[JingsongLi]

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/3290

[BEAM-1498] Use Flink-native side outputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1498

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3290


commit 0a8ad92b9dc5bc90572702592e1b203f4dd909f7
Author: JingsongLi 
Date:   2017-06-04T13:56:10Z

[BEAM-1498] Use Flink-native side outputs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4008

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #92

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Fix race condition when outputting pushed-back elements in 
Flink Runner

[aljoscha.krettek] [BEAM-2401] Update Flink Runner to Flink 1.3.0

[hepei.hp] Flink runner: specify CheckpointingMode through PipelineOptions.

[hepei.hp] Flink runner: specify checkpointTimeout  through PipelineOptions.

--
[...truncated 2.53 MB...]
2017-06-04T12:31:56.012 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/groovy-maven-plugin/2.0/groovy-maven-plugin-2.0.jar
 (48 KB at 3379.0 KB/sec)
2017-06-04T12:31:56.018 [INFO] 
2017-06-04T12:31:56.018 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-04T12:31:56.023 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-04T12:31:56.032 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 160.8 KB/sec)
2017-06-04T12:31:56.036 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-04T12:31:56.045 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 1250.7 KB/sec)
2017-06-04T12:31:56.049 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-04T12:31:56.060 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1166.7 KB/sec)
2017-06-04T12:31:56.064 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-04T12:31:56.072 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 271.6 KB/sec)
2017-06-04T12:31:56.076 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-04T12:31:56.085 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 174.8 KB/sec)
2017-06-04T12:31:56.089 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-04T12:31:56.099 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 525.0 KB/sec)
2017-06-04T12:31:56.103 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-04T12:31:56.110 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 356.4 KB/sec)
2017-06-04T12:31:56.114 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-04T12:31:56.124 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-04T12:31:56.128 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-04T12:31:56.137 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 111.4 KB/sec)
2017-06-04T12:31:56.142 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-04T12:31:56.150 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 212.0 KB/sec)
2017-06-04T12:31:56.156 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-04T12:31:56.166 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 176.4 KB/sec)
2017-06-04T12:31:56.171 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-04T12:31:56.181 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 265.4 KB/sec)
2017-06-04T12:31:56.186 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-04T12:31:56.197 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1601.5 KB/sec)
2017-06-04T12:31:56.201 [INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #42

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Fix race condition when outputting pushed-back elements in 
Flink Runner

[aljoscha.krettek] [BEAM-2401] Update Flink Runner to Flink 1.3.0

[hepei.hp] Flink runner: specify CheckpointingMode through PipelineOptions.

[hepei.hp] Flink runner: specify checkpointTimeout  through PipelineOptions.

--
[...truncated 905.45 KB...]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(1ba556b965f5579f): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #2261

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #42

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Fix race condition when outputting pushed-back elements in 
Flink Runner

[aljoscha.krettek] [BEAM-2401] Update Flink Runner to Flink 1.3.0

[hepei.hp] Flink runner: specify CheckpointingMode through PipelineOptions.

[hepei.hp] Flink runner: specify checkpointTimeout  through PipelineOptions.

--
[...truncated 890.34 KB...]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(d9396822ffb81bee): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2260

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump #172

[Apache Jenkins Server]

See 


--
GitHub pull request #3286 of commit 3be3911e15c58b4020ca0f32770dd99730ae6637, 
no merge conflicts.
Setting status of 3be3911e15c58b4020ca0f32770dd99730ae6637 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/172/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Apache Gearpump Runner ValidatesRunner Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/3286/*:refs/remotes/origin/pr/3286/*
 > git rev-parse refs/remotes/origin/pr/3286/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/3286/merge^{commit} # timeout=10
Checking out Revision 62076d0ca34b005b7eb6058e7cc86e45f93627df 
(refs/remotes/origin/pr/3286/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 62076d0ca34b005b7eb6058e7cc86e45f93627df
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 39241
maven33-agent.jar already up to date
maven33-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_ValidatesRunner_Gearpump] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS 
-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -cp 
/home/jenkins/jenkins-slave/maven33-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven33Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven33-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 39241
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/gearpump -DforkCount=0 
-DvalidatesRunnerPipelineOptions=[ "--runner=TestGearpumpRunner", 
"--streaming=false" ]
2017-06-04T10:07:01.224 [INFO] Error stacktraces are turned on.
2017-06-04T10:07:01.424 [INFO] Scanning for projects...
2017-06-04T10:07:03.742 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
2017-06-04T10:07:04.361 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
 (7 KB at 10.0 KB/sec)
2017-06-04T10:07:04.370 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-06-04T10:07:04.416 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 139.5 KB/sec)
2017-06-04T10:07:04.425 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-06-04T10:07:04.464 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 84.7 KB/sec)
2017-06-04T10:07:04.466 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-06-04T10:07:04.532 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 334.2 KB/sec)
2017-06-04T10:07:04.535 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-06-04T10:07:04.588 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 612.0 KB/sec)
2017-06-04T10:07:04.592 

[GitHub] beam pull request #3289: Flink runner: support MapState in FlinkStateInterna...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/3289

Flink runner: support MapState in FlinkStateInternals.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam flink-map-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3289


commit 071a11f50aaa30075cb6358c41a9ffae1a9a9871
Author: 波特 
Date:   2017-05-18T04:23:20Z

Flink runner: support MapState in FlinkStateInternals.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3245: Flink runner: specify CheckpointingMode and Checkpo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3245


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #3245

[pei]

This closes #3245


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/462335ca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/462335ca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/462335ca

Branch: refs/heads/master
Commit: 462335caf8d148eff8aee86fb00b53960fcd5895
Parents: 1866a01 802f10a
Author: Pei He 
Authored: Sun Jun 4 16:19:36 2017 +0800
Committer: Pei He 
Committed: Sun Jun 4 16:19:36 2017 +0800

--
 .../runners/flink/FlinkPipelineExecutionEnvironment.java |  4 +++-
 .../apache/beam/runners/flink/FlinkPipelineOptions.java  | 11 +++
 2 files changed, 14 insertions(+), 1 deletion(-)
--

[2/3] beam git commit: Flink runner: specify CheckpointingMode through PipelineOptions.

[pei]

Flink runner: specify CheckpointingMode through PipelineOptions.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b8035ae7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b8035ae7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b8035ae7

Branch: refs/heads/master
Commit: b8035ae7ad226bd0261a70fb8e0041e0f07e6dfe
Parents: 1866a01
Author: Pei He 
Authored: Sat May 27 14:41:26 2017 +0800
Committer: Pei He 
Committed: Sun Jun 4 16:18:36 2017 +0800

--
 .../beam/runners/flink/FlinkPipelineExecutionEnvironment.java  | 2 +-
 .../org/apache/beam/runners/flink/FlinkPipelineOptions.java| 6 ++
 2 files changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b8035ae7/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
index 7765a00..98f7c5a 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
@@ -227,7 +227,7 @@ class FlinkPipelineExecutionEnvironment {
   if (checkpointInterval < 1) {
 throw new IllegalArgumentException("The checkpoint interval must be 
positive");
   }
-  flinkStreamEnv.enableCheckpointing(checkpointInterval);
+  flinkStreamEnv.enableCheckpointing(checkpointInterval, 
options.getCheckpointingMode());
   boolean externalizedCheckpoint = 
options.isExternalizedCheckpointsEnabled();
   boolean retainOnCancellation = 
options.getRetainExternalizedCheckpointsOnCancellation();
   if (externalizedCheckpoint) {

http://git-wip-us.apache.org/repos/asf/beam/blob/b8035ae7/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 764fa5f..ee07abb 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -26,6 +26,7 @@ import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.StreamingOptions;
 import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
 
 /**
  * Options which can be used to configure a Flink PipelineRunner.
@@ -70,6 +71,11 @@ public interface FlinkPipelineOptions
   Long getCheckpointingInterval();
   void setCheckpointingInterval(Long interval);
 
+  @Description("The checkpointing mode that defines consistency guarantee.")
+  @Default.Enum("AT_LEAST_ONCE")
+  CheckpointingMode getCheckpointingMode();
+  void setCheckpointingMode(CheckpointingMode mode);
+
   @Description("Sets the number of times that failed tasks are re-executed. "
   + "A value of zero effectively disables fault tolerance. A value of -1 
indicates "
   + "that the system default value (as defined in the configuration) 
should be used.")

[1/3] beam git commit: Flink runner: specify checkpointTimeout through PipelineOptions.

[pei]

Repository: beam
Updated Branches:
  refs/heads/master 1866a0113 -> 462335caf


Flink runner: specify checkpointTimeout  through PipelineOptions.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/802f10af
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/802f10af
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/802f10af

Branch: refs/heads/master
Commit: 802f10afd5d73ba32ad90ba222f2d80216a18a4d
Parents: b8035ae
Author: Pei He 
Authored: Sat May 27 14:59:22 2017 +0800
Committer: Pei He 
Committed: Sun Jun 4 16:18:36 2017 +0800

--
 .../beam/runners/flink/FlinkPipelineExecutionEnvironment.java   | 2 ++
 .../org/apache/beam/runners/flink/FlinkPipelineOptions.java | 5 +
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/802f10af/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
index 98f7c5a..fe5dd87 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
@@ -228,6 +228,8 @@ class FlinkPipelineExecutionEnvironment {
 throw new IllegalArgumentException("The checkpoint interval must be 
positive");
   }
   flinkStreamEnv.enableCheckpointing(checkpointInterval, 
options.getCheckpointingMode());
+  flinkStreamEnv.getCheckpointConfig().setCheckpointTimeout(
+  options.getCheckpointTimeoutMillis());
   boolean externalizedCheckpoint = 
options.isExternalizedCheckpointsEnabled();
   boolean retainOnCancellation = 
options.getRetainExternalizedCheckpointsOnCancellation();
   if (externalizedCheckpoint) {

http://git-wip-us.apache.org/repos/asf/beam/blob/802f10af/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index ee07abb..c255672 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -76,6 +76,11 @@ public interface FlinkPipelineOptions
   CheckpointingMode getCheckpointingMode();
   void setCheckpointingMode(CheckpointingMode mode);
 
+  @Description("The maximum time that a checkpoint may take before being 
discarded.")
+  @Default.Long(20 * 60 * 1000)
+  Long getCheckpointTimeoutMillis();
+  void setCheckpointTimeoutMillis(Long checkpointTimeoutMillis);
+
   @Description("Sets the number of times that failed tasks are re-executed. "
   + "A value of zero effectively disables fault tolerance. A value of -1 
indicates "
   + "that the system default value (as defined in the configuration) 
should be used.")

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4005

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4004

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-567) FlinkRunner pollutes beam-examples-java[8] modules with SLF4J backend dependencies which are not optional (e.g. test/provided/system scoped, or marked optional)

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-567.
-
   Resolution: Fixed
Fix Version/s: 2.1.0

This was fixed by updating to Flink 1.3.0 which doesn't expose the dependency 
anymore.

> FlinkRunner pollutes beam-examples-java[8] modules with SLF4J backend 
> dependencies which are not optional (e.g. test/provided/system scoped, or 
> marked optional)
> 
>
> Key: BEAM-567
> URL: https://issues.apache.org/jira/browse/BEAM-567
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Luke Cwik
>Priority: Minor
> Fix For: 2.1.0
>
>
> Exposed SLF4J backends:
> org.slf4j:slf4j-log4j12:jar:1.7.7:compile
> mvn dependency:tree -pl runners/flink/runner/
> [INFO] 
> 
> [INFO] Building Apache Beam :: Runners :: Flink :: Core 
> 0.3.0-incubating-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ 
> beam-runners-flink_2.10 ---
> [INFO] org.apache.beam:beam-runners-flink_2.10:jar:0.3.0-incubating-SNAPSHOT
> [INFO] +- org.apache.flink:flink-streaming-java_2.10:jar:1.0.3:compile
> [INFO] |  +- org.apache.flink:flink-core:jar:1.0.3:compile
> [INFO] |  +- org.apache.flink:flink-runtime_2.10:jar:1.0.3:compile
> [INFO] |  |  +- io.netty:netty-all:jar:4.0.27.Final:compile
> [INFO] |  |  +- org.javassist:javassist:jar:3.18.2-GA:compile
> [INFO] |  |  +- org.scala-lang:scala-library:jar:2.10.4:compile
> [INFO] |  |  +- com.typesafe.akka:akka-actor_2.10:jar:2.3.7:compile
> [INFO] |  |  |  \- com.typesafe:config:jar:1.2.1:compile
> [INFO] |  |  +- com.typesafe.akka:akka-remote_2.10:jar:2.3.7:compile
> [INFO] |  |  |  +- io.netty:netty:jar:3.8.0.Final:compile
> [INFO] |  |  |  \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
> [INFO] |  |  +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.7:compile
> [INFO] |  |  +- org.clapper:grizzled-slf4j_2.10:jar:1.0.2:compile
> [INFO] |  |  +- com.github.scopt:scopt_2.10:jar:3.2.0:compile
> [INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile
> [INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.0:compile
> [INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.1.0:compile
> [INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
> [INFO] |  |  |  \- jline:jline:jar:0.9.94:compile
> [INFO] |  |  \- com.twitter:chill_2.10:jar:0.7.4:compile
> [INFO] |  | \- com.twitter:chill-java:jar:0.7.4:compile
> [INFO] |  +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] |  +- org.apache.sling:org.apache.sling.commons.json:jar:2.0.6:compile
> [INFO] |  +- org.apache.commons:commons-math3:jar:3.5:compile
> [INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:compile
> [INFO] |  +- org.slf4j:slf4j-api:jar:1.7.14:compile
> [INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.7:compile
> [INFO] |  +- log4j:log4j:jar:1.2.17:compile
> [INFO] |  \- org.apache.flink:force-shading:jar:1.0.3:compile
> [INFO] +- org.apache.flink:flink-java:jar:1.0.3:compile
> [INFO] |  \- org.apache.flink:flink-shaded-hadoop2:jar:1.0.3:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-codec:commons-codec:jar:1.4:compile
> [INFO] | +- commons-io:commons-io:jar:2.4:compile
> [INFO] | +- commons-net:commons-net:jar:3.1:compile
> [INFO] | +- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] | +- javax.servlet:servlet-api:jar:2.5:compile
> [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile
> [INFO] | +- commons-el:commons-el:jar:1.0:runtime
> [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
> [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
> [INFO] | +- commons-lang:commons-lang:jar:2.6:compile
> [INFO] | +- commons-configuration:commons-configuration:jar:1.7:compile
> [INFO] | +- commons-digester:commons-digester:jar:1.8.1:compile
> [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
> [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
> [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] | +- com.jcraft:jsch:jar:0.1.42:compile
> [INFO] | +- org.tukaani:xz:jar:1.0:compile
> [INFO] | +- 
> commons-beanutils:commons-beanutils-bean-collections:jar:1.8.3:compile
> [INFO] | +- commons-daemon:commons-daemon:jar:1.0.13:compile
> [INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
> [INFO] | +- 

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #41

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-2380) Flink Batch Runner does not forward additional outputs to operator

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2380.
--
Resolution: Fixed

> Flink Batch Runner does not forward additional outputs to operator
> --
>
> Key: BEAM-2380
> URL: https://issues.apache.org/jira/browse/BEAM-2380
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.1.0
>
>
> Recent changes to {{SimpleDoFnRunner}} made the output tag validation more 
> strict in {{outputWindowedValue()}}. This surfaced a bug in the Flink Batch 
> runner: it doesn't forward the additional output tags to the 
> {{SimpleDoFnRunner}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2401) Update Flink Runner to Flink 1.3.0

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2401.
--
Resolution: Fixed

> Update Flink Runner to Flink 1.3.0
> --
>
> Key: BEAM-2401
> URL: https://issues.apache.org/jira/browse/BEAM-2401
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> http://flink.apache.org/news/2017/06/01/release-1.3.0.html
> There are a lot of exciting improvements.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2401) Update Flink Runner to Flink 1.3.0

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-2401:
---
Fix Version/s: 2.1.0

> Update Flink Runner to Flink 1.3.0
> --
>
> Key: BEAM-2401
> URL: https://issues.apache.org/jira/browse/BEAM-2401
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> http://flink.apache.org/news/2017/06/01/release-1.3.0.html
> There are a lot of exciting improvements.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #91

[Apache Jenkins Server]

See 


--
[...truncated 2.53 MB...]
2017-06-04T06:30:40.687 [INFO] 
2017-06-04T06:30:40.687 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-04T06:30:40.691 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-04T06:30:40.702 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 131.6 KB/sec)
2017-06-04T06:30:40.708 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-04T06:30:40.718 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 1125.6 KB/sec)
2017-06-04T06:30:40.722 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-04T06:30:40.732 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1283.4 KB/sec)
2017-06-04T06:30:40.737 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-04T06:30:40.747 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 217.3 KB/sec)
2017-06-04T06:30:40.751 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-04T06:30:40.761 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 157.3 KB/sec)
2017-06-04T06:30:40.765 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-04T06:30:40.774 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 583.3 KB/sec)
2017-06-04T06:30:40.777 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-04T06:30:40.788 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 226.8 KB/sec)
2017-06-04T06:30:40.792 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-04T06:30:40.802 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-04T06:30:40.808 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-04T06:30:40.817 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 111.4 KB/sec)
2017-06-04T06:30:40.821 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-04T06:30:40.829 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 212.0 KB/sec)
2017-06-04T06:30:40.834 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-04T06:30:40.843 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 196.0 KB/sec)
2017-06-04T06:30:40.847 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-04T06:30:40.857 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 265.4 KB/sec)
2017-06-04T06:30:40.861 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-04T06:30:40.874 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1355.1 KB/sec)
2017-06-04T06:30:40.878 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-04T06:30:40.889 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 857.1 KB/sec)
2017-06-04T06:30:40.894 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-04T06:30:40.904 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 445.4 KB/sec)
2017-06-04T06:30:40.908 [INFO] Downloading: 

[jira] [Commented] (BEAM-2401) Update Flink Runner to Flink 1.3.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036184#comment-16036184
 ] 

ASF GitHub Bot commented on BEAM-2401:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3284


> Update Flink Runner to Flink 1.3.0
> --
>
> Key: BEAM-2401
> URL: https://issues.apache.org/jira/browse/BEAM-2401
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> http://flink.apache.org/news/2017/06/01/release-1.3.0.html
> There are a lot of exciting improvements.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3284: [BEAM-2401] Update Flink Runner to Flink 1.3.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3284

[aljoscha]

This closes #3284


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1866a011
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1866a011
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1866a011

Branch: refs/heads/master
Commit: 1866a01138ac9095aa7d8dfb1d67c955678b393e
Parents: ef56ea4 fbc6cc5
Author: Aljoscha Krettek 
Authored: Sun Jun 4 08:30:30 2017 +0200
Committer: Aljoscha Krettek 
Committed: Sun Jun 4 08:30:30 2017 +0200

--
 runners/flink/pom.xml   |  2 +-
 .../translation/types/CoderTypeSerializer.java  | 55 
 .../types/EncodedValueSerializer.java   | 18 +--
 .../state/FlinkBroadcastStateInternals.java | 29 +--
 .../streaming/UnboundedSourceWrapperTest.java   | 37 +
 5 files changed, 109 insertions(+), 32 deletions(-)
--

[1/2] beam git commit: [BEAM-2401] Update Flink Runner to Flink 1.3.0

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master ef56ea495 -> 1866a0113


[BEAM-2401] Update Flink Runner to Flink 1.3.0


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fbc6cc59
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fbc6cc59
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fbc6cc59

Branch: refs/heads/master
Commit: fbc6cc59bff93dfcf8676f874870a43eeb228c15
Parents: ef56ea4
Author: JingsongLi 
Authored: Fri Jun 2 11:31:44 2017 +0800
Committer: Aljoscha Krettek 
Committed: Sun Jun 4 08:18:20 2017 +0200

--
 runners/flink/pom.xml   |  2 +-
 .../translation/types/CoderTypeSerializer.java  | 55 
 .../types/EncodedValueSerializer.java   | 18 +--
 .../state/FlinkBroadcastStateInternals.java | 29 +--
 .../streaming/UnboundedSourceWrapperTest.java   | 37 +
 5 files changed, 109 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fbc6cc59/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index fb0a67c..92f95a0 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -31,7 +31,7 @@
   jar
 
   
-1.2.1
+1.3.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/fbc6cc59/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
index e003119..bea562e 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
@@ -24,7 +24,9 @@ import 
org.apache.beam.runners.flink.translation.wrappers.DataOutputViewWrapper;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.flink.api.common.typeutils.CompatibilityResult;
 import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerConfigSnapshot;
 import org.apache.flink.core.memory.DataInputView;
 import org.apache.flink.core.memory.DataOutputView;
 
@@ -129,4 +131,57 @@ public class CoderTypeSerializer extends 
TypeSerializer {
   public int hashCode() {
 return coder.hashCode();
   }
+
+  @Override
+  public TypeSerializerConfigSnapshot snapshotConfiguration() {
+return new CoderTypeSerializerConfigSnapshot<>(coder);
+  }
+
+  @Override
+  public CompatibilityResult 
ensureCompatibility(TypeSerializerConfigSnapshot configSnapshot) {
+if (configSnapshot instanceof CoderTypeSerializerConfigSnapshot) {
+  if (coder.equals(((CoderTypeSerializerConfigSnapshot) 
configSnapshot).coder)) {
+return CompatibilityResult.compatible();
+  }
+}
+return CompatibilityResult.requiresMigration();
+  }
+
+  /**
+   *  TypeSerializerConfigSnapshot of CoderTypeSerializer.
+   */
+  public static class CoderTypeSerializerConfigSnapshot extends 
TypeSerializerConfigSnapshot {
+
+private static final int VERSION = 1;
+private Coder coder;
+
+public CoderTypeSerializerConfigSnapshot(Coder coder) {
+  this.coder = coder;
+}
+
+@Override
+public int getVersion() {
+  return VERSION;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+
+  CoderTypeSerializerConfigSnapshot that = 
(CoderTypeSerializerConfigSnapshot) o;
+
+  return coder != null ? coder.equals(that.coder) : that.coder == null;
+}
+
+@Override
+public int hashCode() {
+  return coder.hashCode();
+}
+  }
+
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/fbc6cc59/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
index c3b9794..c40eb46 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
+++ 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #2256

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #41

[Apache Jenkins Server]

See 


--
[...truncated 876.05 KB...]
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.api-client:google-api-client-java6:jar:1.22.0 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.api-client:google-api-client-jackson2:jar:1.22.0 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 from the shaded 
jar.
2017-06-04\T\06:13:05.579 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-04\T\06:13:05.579 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-06-04\T\06:13:09.999 [INFO] Replacing original artifact with shaded 
artifact.
2017-06-04\T\06:13:09.999 [INFO] Replacing 

 with 

2017-06-04\T\06:13:10.000 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-06-04\T\06:13:10.000 [INFO] Replacing 

 with 

2017-06-04\T\06:13:10.000 [INFO] Dependency-reduced POM written at: 

2017-06-04\T\06:13:10.114 [INFO] 
2017-06-04\T\06:13:10.114 [INFO] --- 
maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
2017-06-04\T\06:13:10.199 [INFO] Failsafe report directory: 

2017-06-04\T\06:13:10.204 [INFO] parallel='all', perCoreThreadCount=true, 
threadCount=4, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-06-04\T\06:13:10.222 [INFO] 
2017-06-04\T\06:13:10.222 [INFO] 
---
2017-06-04\T\06:13:10.222 [INFO]  T E S T S
2017-06-04\T\06:13:10.222 [INFO] 
---
2017-06-04\T\06:18:29.658 [INFO] Running 
org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT
2017-06-04\T\06:18:29.673 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 10.803 s - in 

[GitHub] beam pull request #3285: Fix race condition when outputting pushed-back elem...

[aljoscha]

Github user aljoscha closed the pull request at:

https://github.com/apache/beam/pull/3285


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fix race condition when outputting pushed-back elements in Flink Runner

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master 43c44232d -> ef56ea495


Fix race condition when outputting pushed-back elements in Flink Runner

This affected the Flink Streaming Runner DoFnOperator. The recent fix of
emitting pushed-back data when receiving a watermark on the first input
put the emission at the end of the method. This can cause the emitted
data to become late. The fix is to move the pushed-back element emission
to the start of the method.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d17c0132
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d17c0132
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d17c0132

Branch: refs/heads/master
Commit: d17c013240a14b12992cf00f30e5151c7e97f360
Parents: 43c4423
Author: Aljoscha Krettek 
Authored: Fri Jun 2 15:57:01 2017 +0200
Committer: Aljoscha Krettek 
Committed: Sun Jun 4 08:08:14 2017 +0200

--
 .../wrappers/streaming/DoFnOperator.java  | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d17c0132/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index e473046..594fe0e 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -445,6 +445,15 @@ public class DoFnOperator
 
   @Override
   public void processWatermark1(Watermark mark) throws Exception {
+// We do the check here because we are guaranteed to at least get the +Inf 
watermark on the
+// main input when the job finishes.
+if (currentSideInputWatermark >= 
BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
+  // this means we will never see any more side input
+  // we also do the check here because we might have received the 
side-input MAX watermark
+  // before receiving any main-input data
+  emitAllPushedBackData();
+}
+
 if (keyCoder == null) {
   setCurrentInputWatermark(mark.getTimestamp());
   long potentialOutputWatermark =
@@ -476,15 +485,6 @@ public class DoFnOperator
   }
   pushbackDoFnRunner.finishBundle();
 }
-
-// We do the check here because we are guaranteed to at least get the +Inf 
watermark on the
-// main input when the job finishes.
-if (currentSideInputWatermark >= 
BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
-  // this means we will never see any more side input
-  // we also do the check here because we might have received the 
side-input MAX watermark
-  // before receiving any main-input data
-  emitAllPushedBackData();
-}
   }
 
   @Override

[2/2] beam git commit: This closes #3285

[aljoscha]

This closes #3285


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ef56ea49
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ef56ea49
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ef56ea49

Branch: refs/heads/master
Commit: ef56ea49569f00e098045ec01812b962adfb157d
Parents: 43c4423 d17c013
Author: Aljoscha Krettek 
Authored: Sun Jun 4 08:14:08 2017 +0200
Committer: Aljoscha Krettek 
Committed: Sun Jun 4 08:14:08 2017 +0200

--
 .../wrappers/streaming/DoFnOperator.java  | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)
--