Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4091

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3056) dataflow-runner-integration-tests @ beam-examples-java fail in precommit suite on 2.2.0 branch

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3056:
--
Description: 
[UPDATE]
org.apache.beam.examples.WordCountIT seems to be consistently failing on the 
release branch, see below.

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull


  was:
[UPDATE]

Original descriptions
---
Here's an example of two instances where the issue affected me in the last few 
days.

The test suite consistently made progress, then timed out after 150 min.

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull

What are our options here, besides increasing the timeout?


> dataflow-runner-integration-tests @ beam-examples-java  fail in precommit 
> suite on 2.2.0 branch
> ---
>
> Key: BEAM-3056
> URL: https://issues.apache.org/jira/browse/BEAM-3056
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.2.0
>Reporter: Valentyn Tymofieiev
>Assignee: Reuven Lax
>Priority: Blocker
>
> [UPDATE]
> org.apache.beam.examples.WordCountIT seems to be consistently failing on the 
> release branch, see below.
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3056) dataflow-runner-integration-tests @ beam-examples-java fail in precommit suite on 2.2.0 branch

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3056:
--
Description: 
[UPDATE]
org.apache.beam.examples.WordCountIT seems to be consistently failing on the 
release branch:

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15069/.

  was:
[UPDATE]
org.apache.beam.examples.WordCountIT seems to be consistently failing on the 
release branch, see below.

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull



> dataflow-runner-integration-tests @ beam-examples-java  fail in precommit 
> suite on 2.2.0 branch
> ---
>
> Key: BEAM-3056
> URL: https://issues.apache.org/jira/browse/BEAM-3056
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.2.0
>Reporter: Valentyn Tymofieiev
>Assignee: Reuven Lax
>Priority: Blocker
>
> [UPDATE]
> org.apache.beam.examples.WordCountIT seems to be consistently failing on the 
> release branch:
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15069/.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3056) dataflow-runner-integration-tests @ beam-examples-java fail in precommit suite on 2.2.0 branch

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3056:
--
Description: 
[UPDATE]

Original descriptions
---
Here's an example of two instances where the issue affected me in the last few 
days.

The test suite consistently made progress, then timed out after 150 min.

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull

What are our options here, besides increasing the timeout?

  was:
Here's an example of two instances where the issue affected me in the last few 
days.

The test suite consistently made progress, then timed out after 150 min.

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull

What are our options here, besides increasing the timeout?


> dataflow-runner-integration-tests @ beam-examples-java  fail in precommit 
> suite on 2.2.0 branch
> ---
>
> Key: BEAM-3056
> URL: https://issues.apache.org/jira/browse/BEAM-3056
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.2.0
>Reporter: Valentyn Tymofieiev
>Assignee: Reuven Lax
>Priority: Blocker
>
> [UPDATE]
> Original descriptions
> ---
> Here's an example of two instances where the issue affected me in the last 
> few days.
> The test suite consistently made progress, then timed out after 150 min.
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15044/consoleFull
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15037/consoleFull
> What are our options here, besides increasing the timeout?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-1017) Add RedisIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1017.

   Resolution: Fixed
Fix Version/s: 2.3.0
   2.2.0

> Add RedisIO
> ---
>
> Key: BEAM-1017
> URL: https://issues.apache.org/jira/browse/BEAM-1017
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.2.0, 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

beam git commit: [BEAM-1017] Add RedisIO

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 5601f0ae3 -> e7a8620e0


[BEAM-1017] Add RedisIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e7a8620e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e7a8620e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e7a8620e

Branch: refs/heads/release-2.2.0
Commit: e7a8620e028350c4f48a9f356fb041b6f12e8e34
Parents: 5601f0a
Author: Jean-Baptiste Onofré 
Authored: Wed Mar 22 19:03:00 2017 +0100
Committer: Jean-Baptiste Onofré 
Committed: Tue Oct 17 08:00:17 2017 +0200

--
 sdks/java/io/pom.xml|   1 +
 sdks/java/io/redis/pom.xml  |  90 
 .../io/redis/RedisConnectionConfiguration.java  | 122 +
 .../org/apache/beam/sdk/io/redis/RedisIO.java   | 451 +++
 .../apache/beam/sdk/io/redis/package-info.java  |  22 +
 .../apache/beam/sdk/io/redis/RedisIOTest.java   | 109 +
 6 files changed, 795 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e7a8620e/sdks/java/io/pom.xml
--
diff --git a/sdks/java/io/pom.xml b/sdks/java/io/pom.xml
index 666a2ba..2477868 100644
--- a/sdks/java/io/pom.xml
+++ b/sdks/java/io/pom.xml
@@ -57,6 +57,7 @@
 kinesis
 mongodb
 mqtt
+redis
 solr
 tika
 xml

http://git-wip-us.apache.org/repos/asf/beam/blob/e7a8620e/sdks/java/io/redis/pom.xml
--
diff --git a/sdks/java/io/redis/pom.xml b/sdks/java/io/redis/pom.xml
new file mode 100644
index 000..1aab0f0
--- /dev/null
+++ b/sdks/java/io/redis/pom.xml
@@ -0,0 +1,90 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+
+  4.0.0
+
+  
+org.apache.beam
+beam-sdks-java-io-parent
+2.2.0-SNAPSHOT
+../pom.xml
+  
+
+  beam-sdks-java-io-redis
+  Apache Beam :: SDKs :: Java :: IO :: Redis
+  IO to read and write on a Redis keystore.
+
+  
+
+  org.apache.beam
+  beam-sdks-java-core
+
+
+
+  com.google.guava
+  guava
+
+
+
+  com.google.code.findbugs
+  jsr305
+
+
+
+  redis.clients
+  jedis
+  2.9.0
+
+
+
+
+  com.google.auto.value
+  auto-value
+  provided
+
+
+
+
+  junit
+  junit
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  test
+
+
+  org.slf4j
+  slf4j-jdk14
+  test
+
+
+  org.hamcrest
+  hamcrest-all
+  test
+
+
+  com.github.kstyrc
+  embedded-redis
+  0.6
+  test
+
+  
+
+

http://git-wip-us.apache.org/repos/asf/beam/blob/e7a8620e/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
--
diff --git 
a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
 
b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
new file mode 100644
index 000..efcc77b
--- /dev/null
+++ 
b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.redis;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+
+import java.io.Serializable;
+
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.transforms.display.DisplayData;
+
+import redis.clients.jedis.Jedis;
+import redis.clients.jedis.Protocol;
+
+/**
+ * {@code RedisConnectionConfiguration} describes and wraps a 
connectionConfiguration to Redis
+ * server or cluster.
+ */
+@AutoValue
+public abstract class RedisConnectionConfiguration implements Serializable {
+
+  a

[jira] [Commented] (BEAM-3011) Pin Runner harness container image in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207058#comment-16207058
 ] 

ASF GitHub Bot commented on BEAM-3011:
--

GitHub user tvalentyn reopened a pull request:

https://github.com/apache/beam/pull/3970

Pin runner harness container image for Dataflow distributions of BEAM…

… Python SDK

This fixes https://issues.apache.org/jira/browse/BEAM-3011

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam 
pin_runner_harness_cherrypick_onto_release

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3970.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3970


commit f260c284cd48c2b326baf480ebe7f6f99cd42e81
Author: Valentyn Tymofieiev 
Date:   2017-10-09T22:07:56Z

Pin runner harness container image for Dataflow distributions of BEAM 
Python SDK

commit ce388bc51fbb6eb704db210d33a027e65bcdc6f9
Author: Valentyn Tymofieiev 
Date:   2017-10-10T19:45:25Z

Make linter happy.

commit b00473098dec8fe091bc862ea69b8bdcd098f1cf
Author: Henning Rohde 
Date:   2017-10-09T22:28:25Z

[BEAM-2600] Add minimal python SDK harness container

commit 999c8d633e3db7cbb8717ef80f0472c8d88afa57
Author: Valentyn Tymofieiev 
Date:   2017-10-12T16:07:50Z

Use 2.2.0 version instead of 2.3.0 for container Maven project on the 
release branch. This reverts eeb4d73.

commit 1e0a6bcd72b1aeea93d8644e8776ca762d175f47
Author: Valentyn Tymofieiev 
Date:   2017-10-12T20:10:00Z

Use tox instead of detox. Looks like we reverted to tox on the master 
branch.




> Pin Runner harness container image in Python SDK
> 
>
> Key: BEAM-3011
> URL: https://issues.apache.org/jira/browse/BEAM-3011
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
>  Labels: portability
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1017) Add RedisIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207056#comment-16207056
 ] 

ASF GitHub Bot commented on BEAM-1017:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1687


> Add RedisIO
> ---
>
> Key: BEAM-1017
> URL: https://issues.apache.org/jira/browse/BEAM-1017
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3970: Pin runner harness container image for Dataflow dis...

[tvalentyn]

Github user tvalentyn closed the pull request at:

https://github.com/apache/beam/pull/3970


---

[GitHub] beam pull request #3970: Pin runner harness container image for Dataflow dis...

[tvalentyn]

GitHub user tvalentyn reopened a pull request:

https://github.com/apache/beam/pull/3970

Pin runner harness container image for Dataflow distributions of BEAM…

… Python SDK

This fixes https://issues.apache.org/jira/browse/BEAM-3011

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam 
pin_runner_harness_cherrypick_onto_release

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3970.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3970


commit f260c284cd48c2b326baf480ebe7f6f99cd42e81
Author: Valentyn Tymofieiev 
Date:   2017-10-09T22:07:56Z

Pin runner harness container image for Dataflow distributions of BEAM 
Python SDK

commit ce388bc51fbb6eb704db210d33a027e65bcdc6f9
Author: Valentyn Tymofieiev 
Date:   2017-10-10T19:45:25Z

Make linter happy.

commit b00473098dec8fe091bc862ea69b8bdcd098f1cf
Author: Henning Rohde 
Date:   2017-10-09T22:28:25Z

[BEAM-2600] Add minimal python SDK harness container

commit 999c8d633e3db7cbb8717ef80f0472c8d88afa57
Author: Valentyn Tymofieiev 
Date:   2017-10-12T16:07:50Z

Use 2.2.0 version instead of 2.3.0 for container Maven project on the 
release branch. This reverts eeb4d73.

commit 1e0a6bcd72b1aeea93d8644e8776ca762d175f47
Author: Valentyn Tymofieiev 
Date:   2017-10-12T20:10:00Z

Use tox instead of detox. Looks like we reverted to tox on the master 
branch.




---

[jira] [Commented] (BEAM-3011) Pin Runner harness container image in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207057#comment-16207057
 ] 

ASF GitHub Bot commented on BEAM-3011:
--

Github user tvalentyn closed the pull request at:

https://github.com/apache/beam/pull/3970


> Pin Runner harness container image in Python SDK
> 
>
> Key: BEAM-3011
> URL: https://issues.apache.org/jira/browse/BEAM-3011
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
>  Labels: portability
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #1687: [BEAM-1017] Add RedisIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1687


---

[1/2] beam git commit: [BEAM-1017] Add RedisIO

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master 4b502bf71 -> 9524cbd88


[BEAM-1017] Add RedisIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d8ab6cb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d8ab6cb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d8ab6cb

Branch: refs/heads/master
Commit: 0d8ab6cbbc762dd9f9be1b3e9a26b6c9d0bb6dc3
Parents: 4b502bf
Author: Jean-Baptiste Onofré 
Authored: Wed Mar 22 19:03:00 2017 +0100
Committer: Jean-Baptiste Onofré 
Committed: Tue Oct 17 07:37:14 2017 +0200

--
 sdks/java/io/pom.xml|   1 +
 sdks/java/io/redis/pom.xml  |  90 
 .../io/redis/RedisConnectionConfiguration.java  | 122 +
 .../org/apache/beam/sdk/io/redis/RedisIO.java   | 451 +++
 .../apache/beam/sdk/io/redis/package-info.java  |  22 +
 .../apache/beam/sdk/io/redis/RedisIOTest.java   | 109 +
 6 files changed, 795 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0d8ab6cb/sdks/java/io/pom.xml
--
diff --git a/sdks/java/io/pom.xml b/sdks/java/io/pom.xml
index 49eb796..99936a2 100644
--- a/sdks/java/io/pom.xml
+++ b/sdks/java/io/pom.xml
@@ -57,6 +57,7 @@
 kinesis
 mongodb
 mqtt
+redis
 solr
 tika
 xml

http://git-wip-us.apache.org/repos/asf/beam/blob/0d8ab6cb/sdks/java/io/redis/pom.xml
--
diff --git a/sdks/java/io/redis/pom.xml b/sdks/java/io/redis/pom.xml
new file mode 100644
index 000..d89e627
--- /dev/null
+++ b/sdks/java/io/redis/pom.xml
@@ -0,0 +1,90 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+
+  4.0.0
+
+  
+org.apache.beam
+beam-sdks-java-io-parent
+2.3.0-SNAPSHOT
+../pom.xml
+  
+
+  beam-sdks-java-io-redis
+  Apache Beam :: SDKs :: Java :: IO :: Redis
+  IO to read and write on a Redis keystore.
+
+  
+
+  org.apache.beam
+  beam-sdks-java-core
+
+
+
+  com.google.guava
+  guava
+
+
+
+  com.google.code.findbugs
+  jsr305
+
+
+
+  redis.clients
+  jedis
+  2.9.0
+
+
+
+
+  com.google.auto.value
+  auto-value
+  provided
+
+
+
+
+  junit
+  junit
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  test
+
+
+  org.slf4j
+  slf4j-jdk14
+  test
+
+
+  org.hamcrest
+  hamcrest-all
+  test
+
+
+  com.github.kstyrc
+  embedded-redis
+  0.6
+  test
+
+  
+
+

http://git-wip-us.apache.org/repos/asf/beam/blob/0d8ab6cb/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
--
diff --git 
a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
 
b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
new file mode 100644
index 000..efcc77b
--- /dev/null
+++ 
b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.redis;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+
+import java.io.Serializable;
+
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.transforms.display.DisplayData;
+
+import redis.clients.jedis.Jedis;
+import redis.clients.jedis.Protocol;
+
+/**
+ * {@code RedisConnectionConfiguration} describes and wraps a 
connectionConfiguration to Redis
+ * server or cluster.
+ */
+@AutoValue
+public abstract class RedisConnectionConfiguration implements Serializable {
+
+  abstract String

[2/2] beam git commit: [BEAM-1017] This closes #1687

[jbonofre]

[BEAM-1017] This closes #1687


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9524cbd8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9524cbd8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9524cbd8

Branch: refs/heads/master
Commit: 9524cbd88bba33d7fe01691c791f19fbe6239464
Parents: 4b502bf 0d8ab6c
Author: Jean-Baptiste Onofré 
Authored: Tue Oct 17 07:38:06 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue Oct 17 07:38:06 2017 +0200

--
 sdks/java/io/pom.xml|   1 +
 sdks/java/io/redis/pom.xml  |  90 
 .../io/redis/RedisConnectionConfiguration.java  | 122 +
 .../org/apache/beam/sdk/io/redis/RedisIO.java   | 451 +++
 .../apache/beam/sdk/io/redis/package-info.java  |  22 +
 .../apache/beam/sdk/io/redis/RedisIOTest.java   | 109 +
 6 files changed, 795 insertions(+)
--

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5048

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206982#comment-16206982
 ] 

ASF GitHub Bot commented on BEAM-3066:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4004

[BEAM-3066] Add standalone version of seed job

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jasonkuster 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam seed-job-standalone

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4004.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4004


commit daa3c268aba642daea5713434f086eb482832c34
Author: Kenneth Knowles 
Date:   2017-10-17T04:20:52Z

Add standalone version of seed job




> Jenkins seed job should be resilient and independent of other jobs' 
> configurations
> --
>
> Key: BEAM-3066
> URL: https://issues.apache.org/jira/browse/BEAM-3066
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3066:
-

 Summary: Jenkins seed job should be resilient and independent of 
other jobs' configurations
 Key: BEAM-3066
 URL: https://issues.apache.org/jira/browse/BEAM-3066
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4004: [BEAM-3066] Add standalone version of seed job

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4004

[BEAM-3066] Add standalone version of seed job

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jasonkuster 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam seed-job-standalone

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4004.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4004


commit daa3c268aba642daea5713434f086eb482832c34
Author: Kenneth Knowles 
Date:   2017-10-17T04:20:52Z

Add standalone version of seed job




---

[jira] [Resolved] (BEAM-409) Incorrect use of Math.ceil in ApproximateQuantiles

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-409.
--
   Resolution: Fixed
Fix Version/s: 2.3.0

> Incorrect use of Math.ceil in ApproximateQuantiles
> --
>
> Key: BEAM-409
> URL: https://issues.apache.org/jira/browse/BEAM-409
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Daniel Oliveira
>Priority: Minor
>  Labels: findbugs, newbie, starter
> Fix For: 2.3.0
>
>
> [FindBugs 
> ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L274]:
>  Integral value cast to double and then passed to Math.ceil
> Applies to: 
> [ApproximateQuanitles|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L349].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-3051) no beta on gcloud

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3051.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> no beta on gcloud
> -
>
> Key: BEAM-3051
> URL: https://issues.apache.org/jira/browse/BEAM-3051
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
>Reporter: Elliotte Rusty Harold
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: 2.3.0
>
>
> This message is showing up in my console:
> INFO: To cancel the job using the 'gcloud' tool, run:
> > gcloud beta dataflow jobs --project=dataflowproject-169218 cancel 
> > 2017-10-12_13_56_47-17384984791171037732
> This message is incorrect and does not work. The correct command is
>  gcloud dataflow jobs --project=dataflowproject-169218 cancel 
> 2017-10-12_13_56_47-17384984791171037732
> I.e. you need to remove the beta from the gcloud dataflow command. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3051) no beta on gcloud

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206965#comment-16206965
 ] 

ASF GitHub Bot commented on BEAM-3051:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3989


> no beta on gcloud
> -
>
> Key: BEAM-3051
> URL: https://issues.apache.org/jira/browse/BEAM-3051
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
>Reporter: Elliotte Rusty Harold
>Assignee: Thomas Groh
>Priority: Minor
>
> This message is showing up in my console:
> INFO: To cancel the job using the 'gcloud' tool, run:
> > gcloud beta dataflow jobs --project=dataflowproject-169218 cancel 
> > 2017-10-12_13_56_47-17384984791171037732
> This message is incorrect and does not work. The correct command is
>  gcloud dataflow jobs --project=dataflowproject-169218 cancel 
> 2017-10-12_13_56_47-17384984791171037732
> I.e. you need to remove the beta from the gcloud dataflow command. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3989: [BEAM-3051] Remove "beta" from the Cancel Command P...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3989


---

[2/2] beam git commit: Remove "beta" from the Cancel Command Prefix

[tgroh]

Remove "beta" from the Cancel Command Prefix


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2af8b07c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2af8b07c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2af8b07c

Branch: refs/heads/master
Commit: 2af8b07c45869aceab09eaa966570ae1029559ed
Parents: 52ad6f1
Author: Thomas Groh 
Authored: Thu Oct 12 17:06:43 2017 -0700
Committer: Thomas Groh 
Committed: Mon Oct 16 20:47:53 2017 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2af8b07c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index 780a979..cf46406 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -46,7 +46,7 @@ import org.slf4j.LoggerFactory;
  */
 public class MonitoringUtil {
 
-  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud beta dataflow";
+  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud dataflow";
   private static final String ENDPOINT_OVERRIDE_ENV_VAR =
   "CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW";
 

http://git-wip-us.apache.org/repos/asf/beam/blob/2af8b07c/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
index c048776..4991982 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
@@ -121,7 +121,7 @@ public class MonitoringUtilTest {
 options.setProject(PROJECT_ID);
 options.setGcpCredential(new TestCredential());
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
-assertEquals("gcloud beta dataflow jobs --project=someProject cancel 
1234", cancelCommand);
+assertEquals("gcloud dataflow jobs --project=someProject cancel 1234", 
cancelCommand);
   }
 
   @Test
@@ -135,7 +135,7 @@ public class MonitoringUtilTest {
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
 assertEquals(
 
"CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW=https://dataflow.googleapis.com/v0neverExisted/
 "
-+ "gcloud beta dataflow jobs --project=someProject cancel 1234",
++ "gcloud dataflow jobs --project=someProject cancel 1234",
 cancelCommand);
   }
 

[1/2] beam git commit: This closes #3989

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 52ad6f12a -> 4b502bf71


This closes #3989


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4b502bf7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4b502bf7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4b502bf7

Branch: refs/heads/master
Commit: 4b502bf7160a1ef872b31b47c1ae24f56a0cf9ea
Parents: 52ad6f1 2af8b07
Author: Thomas Groh 
Authored: Mon Oct 16 20:47:53 2017 -0700
Committer: Thomas Groh 
Committed: Mon Oct 16 20:47:53 2017 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--

[jira] [Resolved] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2498.
---
Resolution: Fixed

> Dataflow runner should shade Runner/Fn API protos
> -
>
> Key: BEAM-2498
> URL: https://issues.apache.org/jira/browse/BEAM-2498
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.3.0
>
>
> Just checked, and runners-core-construction is shaded but not the Runner API 
> protos. There may be a technical reason this cannot be done trivially, but we 
> need to work at it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206953#comment-16206953
 ] 

Kenneth Knowles commented on BEAM-2498:
---

In the `beam-model-*` refactor, the needed shading was added. I just confirmed 
that the runner shades the model.

> Dataflow runner should shade Runner/Fn API protos
> -
>
> Key: BEAM-2498
> URL: https://issues.apache.org/jira/browse/BEAM-2498
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.3.0
>
>
> Just checked, and runners-core-construction is shaded but not the Runner API 
> protos. There may be a technical reason this cannot be done trivially, but we 
> need to work at it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206953#comment-16206953
 ] 

Kenneth Knowles edited comment on BEAM-2498 at 10/17/17 3:17 AM:
-

In the beam model refactor, the needed shading was added. I just confirmed that 
the runner shades the model.


was (Author: kenn):
In the `beam-model-*` refactor, the needed shading was added. I just confirmed 
that the runner shades the model.

> Dataflow runner should shade Runner/Fn API protos
> -
>
> Key: BEAM-2498
> URL: https://issues.apache.org/jira/browse/BEAM-2498
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.3.0
>
>
> Just checked, and runners-core-construction is shaded but not the Runner API 
> protos. There may be a technical reason this cannot be done trivially, but we 
> need to work at it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3049) Java SDK Harness bundles non-relocated code, including Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206940#comment-16206940
 ] 

ASF GitHub Bot commented on BEAM-3049:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4003

[BEAM-3049] Relocate everything shaded by Java SDK harness

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @herohde 

Verified that everything in the shaded jar begins with 
`org.apache.beam.fn.harness` which is the current namespace adopted by the SDK 
harness code.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam harness-shading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4003


commit 008909392b5b4e7227e07f4a11e13d799100fd46
Author: Kenneth Knowles 
Date:   2017-10-17T02:51:50Z

Relocate everything shaded by Java SDK harness




> Java SDK Harness bundles non-relocated code, including Dataflow runner
> --
>
> Key: BEAM-3049
> URL: https://issues.apache.org/jira/browse/BEAM-3049
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This causes a problem if something depends on the harness but does not want 
> the harness's copy of its dependencies. I know we intend to break the 
> dependency on the Dataflow runner. It also bundles a couple other things 
> unshaded.
> Mostly, the harness should be executed entirely containerized so it doesn't 
> matter, in which case there's no need to relocate anything, and bundling is 
> just a convenience. But we should have a clear policy that we adhere to. 
> Either it is a library and should have good hygeine, or if it doesn't have 
> good hygeine it must not be used as a library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4003: [BEAM-3049] Relocate everything shaded by Java SDK ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4003

[BEAM-3049] Relocate everything shaded by Java SDK harness

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @herohde 

Verified that everything in the shaded jar begins with 
`org.apache.beam.fn.harness` which is the current namespace adopted by the SDK 
harness code.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam harness-shading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4003


commit 008909392b5b4e7227e07f4a11e13d799100fd46
Author: Kenneth Knowles 
Date:   2017-10-17T02:51:50Z

Relocate everything shaded by Java SDK harness




---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #4089

[Apache Jenkins Server]

See 


Changes:

[tgroh] [BEAM-409] Avoiding integer division in ceil

--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 52ad6f12a95219f5318bcb7230068384b22c9d7a (origin/master)
Commit message: "This closes #3861"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 52ad6f12a95219f5318bcb7230068384b22c9d7a
 > git rev-list 8ff9c003f407a8db7941a071280e16c603732f92 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 34866
maven33-agent.jar already up to date
maven33-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_ValidatesRunner_Flink] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS 
-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -cp 
/home/jenkins/jenkins-slave/maven33-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven33Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven33-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 34866
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/flink -Plocal-validates-runner-tests 
-Pvalidates-runner-tests
2017-10-17T02:46:29.103 [INFO] Error stacktraces are turned on.
2017-10-17T02:46:29.388 [INFO] Scanning for projects...
2017-10-17T02:46:31.583 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
2017-10-17T02:46:32.830 [ERROR] [ERROR] Some problems were encountered while 
processing the POMs:
[ERROR] Unresolveable build extension: Plugin 
kr.motd.maven:os-maven-plugin:1.5.0.Final or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
kr.motd.maven:os-maven-plugin:jar:1.5.0.Final @ 
 @ 
2017-10-17T02:46:32.832 [ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were 
encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin 
kr.motd.maven:os-maven-plugin:1.5.0.Final or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
kr.motd.maven:os-maven-plugin:jar:1.5.0.Final @ 

at 
org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:422)
at 
org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:419)
at 
org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:410)
at 
org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:83)
at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:491)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:219)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodA

[jira] [Commented] (BEAM-409) Incorrect use of Math.ceil in ApproximateQuantiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206935#comment-16206935
 ] 

ASF GitHub Bot commented on BEAM-409:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3861


> Incorrect use of Math.ceil in ApproximateQuantiles
> --
>
> Key: BEAM-409
> URL: https://issues.apache.org/jira/browse/BEAM-409
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Daniel Oliveira
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L274]:
>  Integral value cast to double and then passed to Math.ceil
> Applies to: 
> [ApproximateQuanitles|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L349].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3861: [BEAM-409] Fixing incorrect use of Math.ceil in App...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3861


---

[1/2] beam git commit: This closes #3861

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 8ff9c003f -> 52ad6f12a


This closes #3861


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/52ad6f12
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/52ad6f12
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/52ad6f12

Branch: refs/heads/master
Commit: 52ad6f12a95219f5318bcb7230068384b22c9d7a
Parents: 8ff9c00 2f0a53d
Author: Thomas Groh 
Authored: Mon Oct 16 19:43:54 2017 -0700
Committer: Thomas Groh 
Committed: Mon Oct 16 19:43:54 2017 -0700

--
 .../src/main/resources/beam/findbugs-filter.xml |   8 +-
 .../sdk/transforms/ApproximateQuantiles.java|  10 +-
 .../transforms/ApproximateQuantilesTest.java| 528 +++
 3 files changed, 326 insertions(+), 220 deletions(-)
--

[2/2] beam git commit: [BEAM-409] Avoiding integer division in ceil

[tgroh]

[BEAM-409] Avoiding integer division in ceil

Casting the denominator in this division in the ceil call to a float to
avoid an incorrect integer division causing a bug.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f0a53d7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f0a53d7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f0a53d7

Branch: refs/heads/master
Commit: 2f0a53d7519c3ad556901cf99aad0bbf2b206f3e
Parents: 8ff9c00
Author: Daniel Oliveira 
Authored: Fri Sep 15 15:32:49 2017 -0700
Committer: Thomas Groh 
Committed: Mon Oct 16 19:43:54 2017 -0700

--
 .../src/main/resources/beam/findbugs-filter.xml |   8 +-
 .../sdk/transforms/ApproximateQuantiles.java|  10 +-
 .../transforms/ApproximateQuantilesTest.java| 528 +++
 3 files changed, 326 insertions(+), 220 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2f0a53d7/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index e54cd0b..bf10571 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -314,13 +314,7 @@
 
 
 
-
-  
-  
-
-
-
-
+
   
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/2f0a53d7/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
index e952af2..ff37024 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
@@ -341,7 +341,7 @@ public class ApproximateQuantiles {
 b++;
   }
   b--;
-  int k = Math.max(2, (int) Math.ceil(maxNumElements / (1 << (b - 1;
+  int k = Math.max(2, (int) Math.ceil(maxNumElements / (float) (1 << (b - 
1;
   return new ApproximateQuantilesCombineFn(
   numQuantiles, compareFn, k, b, maxNumElements);
 }
@@ -366,6 +366,14 @@ public class ApproximateQuantiles {
   .add(DisplayData.item("comparer", compareFn.getClass())
 .withLabel("Record Comparer"));
 }
+
+int getNumBuffers() {
+  return numBuffers;
+}
+
+int getBufferSize() {
+  return bufferSize;
+}
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/beam/blob/2f0a53d7/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
index e180833..2657e07 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
@@ -21,10 +21,13 @@ import static 
org.apache.beam.sdk.testing.CombineFnTester.testCombineFn;
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
+import static org.junit.Assert.assertEquals;
 
+import com.google.common.collect.Lists;
 import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.Comparator;
 import java.util.List;
 import org.apache.beam.sdk.Pipeline;
@@ -41,270 +44,371 @@ import org.apache.beam.sdk.values.PCollection;
 import org.hamcrest.CoreMatchers;
 import org.hamcrest.Description;
 import org.hamcrest.Matcher;
+import org.hamcrest.Matchers;
 import org.hamcrest.TypeSafeDiagnosingMatcher;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
+import org.junit.experimental.runners.Enclosed;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
 
 /**
  * Tests for {@link ApproximateQuantiles}.
  */
-@RunWith(JUnit4.class)
+@RunWith(Enclosed.class)
 public class ApproximateQuantilesTest {
 
-  static final List> TABLE = Arrays.asList(
-  KV.of("a", 1),
-  KV.of("a", 2),
-  KV.of("a", 3),
-  KV.of("b", 1

[jira] [Commented] (BEAM-3065) Python SDK installation is broken for Windows

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206924#comment-16206924
 ] 

Chamikara Jayalath commented on BEAM-3065:
--

cc: [~altay] [~charleschen]

> Python SDK installation is broken for Windows
> -
>
> Key: BEAM-3065
> URL: https://issues.apache.org/jira/browse/BEAM-3065
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Chamikara Jayalath
>Assignee: Robert Bradshaw
>Priority: Blocker
>
> "python setup.py sdist" and "python setup.py install" raises following error 
> for Windows. Failure happens when running gen_protos.py. Robert, seems like 
> you added this script. Can you take a look ?
> C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py:48:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "c:\python27\Lib\multiprocessing\forking.py", line 380, in main
> prepare(preparation_data)
>   File "c:\python27\Lib\multiprocessing\forking.py", line 510, in prepare
> '__parents_main__', file, path_name, etc
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
>  line 203, in 
> 'test': generate_protos_first(test),
>   File "c:\python27\Lib\distutils\core.py", line 151, in setup
> dist.run_commands()
>   File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
>  line 67, in run
> self.do_egg_install()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
>  line 109, in do_egg_install
> self.run_command('bdist_egg')
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
>  line 169, in run
> cmd = self.call_command('install_lib', warn_dir=0)
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
>  line 155, in call_command
> self.run_command(cmdname)
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install_lib.py",
>  line 11, in run
> self.build()
>   File "c:\python27\Lib\distutils\command\install_lib.py", line 109, in build
> self.run_command('build_py')
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
>  line 143, in run
> gen_protos.generate_proto_files()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py",
>  line 84, in generate_proto_files
> p.start()
>   File "c:\python27\Lib\multiprocessing\process.py", line 130, in start
> self._popen = Popen(self)
>   File "c:\python27\Lib\multiprocessing\forking.py", line 258, in __init__
> cmd = get_command_line() + [rhandle]
>   File "c:\python27\Lib\multiprocessing\forking.py", line 358, in 
> get_command_line
> is not going to be frozen to produce a Windows executable.''')
> RuntimeError:
> Attempt to start a new process before the current process
> has finished its bootstrapping phase.
> This probably means that you are on Windows and you have
> forgotten to use the proper idiom in the main module:
> if __name__ == '__main__':
> freeze_support()
> ...
> The "freeze_support()" line can be omitted if the program
> is not going to be frozen to produce a Windows executable.
> Traceback (most recent call last):
>   File "setup.py", line 203, in 
> 'test': generate_protos_first(test),
>   File "c:\python27\Lib\distutils\core.py", line 151, in setup
> dist.run_commands()
>   File "c:

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4177

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5045

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-3063] Improve VoidCoder structural value to use a single shared

--
[...truncated 2.24 MB...]
2017-10-17T02:05:38.450 [INFO] 
2017-10-17T02:05:38.450 [INFO] --- maven-dependency-plugin:3.0.1:analyze-only 
(default) @ beam-runners-gearpump ---
2017-10-17T02:05:38.668 [INFO] No dependency problems found
2017-10-17T02:05:38.826 [INFO] 
2017-10-17T02:05:38.826 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-runners-gearpump ---
2017-10-17T02:05:38.830 [INFO] Installing 

 to 

2017-10-17T02:05:38.855 [INFO] Installing 

 to 

2017-10-17T02:05:38.856 [INFO] Installing 

 to 

2017-10-17T02:05:38.858 [INFO] Installing 

 to 

2017-10-17T02:05:38.859 [INFO] Installing 

 to 

2017-10-17T02:05:38.860 [INFO] Installing 

 to 

2017-10-17T02:05:38.862 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-10-17T02:05:40.382 [INFO]  
   
2017-10-17T02:05:40.382 [INFO] 

2017-10-17T02:05:40.382 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Aggregated Javadoc
2017-10-17T02:05:40.382 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-17T02:05:40.382 [INFO] 

2017-10-17T02:05:40.642 [INFO]  
   
2017-10-17T02:05:40.642 [INFO] 

2017-10-17T02:05:40.642 [INFO] Building Apache Beam :: Parent 2.3.0-SNAPSHOT
2017-10-17T02:05:40.642 [INFO] 

2017-10-17T02:05:40.643 [WARNING] The POM for 
org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency 
information available
2017-10-17T02:05:40.644 [WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
2017-10-17T02:05:40.644 [INFO] 
2017-10-17T02:05:40.644 [INFO] --- coveralls-maven-plugin:4.3.0:report 
(default-cli) @ beam-parent ---
2017-10-17T02:05:40.658 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.pom
2017-10-

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #2619

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3065) Python SDK installation is broken for Windows

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3065:


 Summary: Python SDK installation is broken for Windows
 Key: BEAM-3065
 URL: https://issues.apache.org/jira/browse/BEAM-3065
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Affects Versions: 2.2.0
Reporter: Chamikara Jayalath
Assignee: Robert Bradshaw
Priority: Blocker


"python setup.py sdist" and "python setup.py install" raises following error 
for Windows. Failure happens when running gen_protos.py. Robert, seems like you 
added this script. Can you take a look ?

C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py:48:
 UserWarning: Installing grpcio-tools is recommended for development.
  warnings.warn('Installing grpcio-tools is recommended for development.')
Traceback (most recent call last):
  File "", line 1, in 
  File "c:\python27\Lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
  File "c:\python27\Lib\multiprocessing\forking.py", line 510, in prepare
'__parents_main__', file, path_name, etc
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
 line 203, in 
'test': generate_protos_first(test),
  File "c:\python27\Lib\distutils\core.py", line 151, in setup
dist.run_commands()
  File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
  File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
 line 67, in run
self.do_egg_install()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
 line 109, in do_egg_install
self.run_command('bdist_egg')
  File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
  File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
 line 169, in run
cmd = self.call_command('install_lib', warn_dir=0)
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
 line 155, in call_command
self.run_command(cmdname)
  File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
  File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install_lib.py",
 line 11, in run
self.build()
  File "c:\python27\Lib\distutils\command\install_lib.py", line 109, in build
self.run_command('build_py')
  File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
  File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
 line 143, in run
gen_protos.generate_proto_files()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py",
 line 84, in generate_proto_files
p.start()
  File "c:\python27\Lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
  File "c:\python27\Lib\multiprocessing\forking.py", line 258, in __init__
cmd = get_command_line() + [rhandle]
  File "c:\python27\Lib\multiprocessing\forking.py", line 358, in 
get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.

This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:

if __name__ == '__main__':
freeze_support()
...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
Traceback (most recent call last):
  File "setup.py", line 203, in 
'test': generate_protos_first(test),
  File "c:\python27\Lib\distutils\core.py", line 151, in setup
dist.run_commands()
  File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
  File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
 line 67, in run
self.do_egg_install()
  File 
"C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-pack

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #2618

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3063.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.3.0
>
>
> VoidCoder structuralValue always performs an encode on something which is 
> effectively zero bytes instead of returning a trivial object which is always 
> equals to itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206820#comment-16206820
 ] 

ASF GitHub Bot commented on BEAM-3063:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4000


> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.3.0
>
>
> VoidCoder structuralValue always performs an encode on something which is 
> effectively zero bytes instead of returning a trivial object which is always 
> equals to itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4000: [BEAM-3063] Improve VoidCoder structural value to u...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4000


---

[2/2] beam git commit: [BEAM-3063] Improve VoidCoder structural value to use a single shared instance.

[lcwik]

[BEAM-3063] Improve VoidCoder structural value to use a single shared instance.

 This closes #4000


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8ff9c003
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8ff9c003
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8ff9c003

Branch: refs/heads/master
Commit: 8ff9c003f407a8db7941a071280e16c603732f92
Parents: 3633c40 380e995
Author: Luke Cwik 
Authored: Mon Oct 16 17:13:54 2017 -0700
Committer: Luke Cwik 
Committed: Mon Oct 16 17:13:54 2017 -0700

--
 .../main/java/org/apache/beam/sdk/coders/VoidCoder.java  | 10 +++---
 .../java/org/apache/beam/sdk/coders/VoidCoderTest.java   | 11 ++-
 2 files changed, 13 insertions(+), 8 deletions(-)
--

[1/2] beam git commit: [BEAM-3063] Improve VoidCoder structural value to use a single shared instance.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 3633c40b9 -> 8ff9c003f


[BEAM-3063] Improve VoidCoder structural value to use a single shared instance.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/380e995c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/380e995c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/380e995c

Branch: refs/heads/master
Commit: 380e995c0f82bf3269a4d98abd751d000a2815d5
Parents: 3633c40
Author: Luke Cwik 
Authored: Mon Oct 16 13:56:04 2017 -0700
Committer: Luke Cwik 
Committed: Mon Oct 16 17:13:30 2017 -0700

--
 .../main/java/org/apache/beam/sdk/coders/VoidCoder.java  | 10 +++---
 .../java/org/apache/beam/sdk/coders/VoidCoderTest.java   | 11 ++-
 2 files changed, 13 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/380e995c/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/VoidCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/VoidCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/VoidCoder.java
index 3e1ff7f..0e2236e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/VoidCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/VoidCoder.java
@@ -34,6 +34,7 @@ public class VoidCoder extends AtomicCoder {
 
   private static final VoidCoder INSTANCE = new VoidCoder();
   private static final TypeDescriptor TYPE_DESCRIPTOR = new 
TypeDescriptor() {};
+  private static final Object STRUCTURAL_VOID_VALUE = new Object();
 
   private VoidCoder() {}
 
@@ -51,14 +52,9 @@ public class VoidCoder extends AtomicCoder {
   @Override
   public void verifyDeterministic() {}
 
-  /**
-   * {@inheritDoc}
-   *
-   * @return  {@code true}. {@link VoidCoder} is (vacuously) injective.
-   */
   @Override
-  public boolean consistentWithEquals() {
-return true;
+  public Object structuralValue(Void value) {
+return STRUCTURAL_VOID_VALUE;
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/beam/blob/380e995c/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java
index e618dbb..4e0f1b7 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java
@@ -18,6 +18,8 @@
 package org.apache.beam.sdk.coders;
 
 import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertSame;
 import static org.junit.Assert.assertThat;
 
 import org.apache.beam.sdk.values.TypeDescriptor;
@@ -30,11 +32,18 @@ import org.junit.runners.JUnit4;
  */
 @RunWith(JUnit4.class)
 public class VoidCoderTest {
-
   private static final Coder TEST_CODER = VoidCoder.of();
 
   @Test
   public void testEncodedTypeDescriptor() throws Exception {
 assertThat(TEST_CODER.getEncodedTypeDescriptor(), 
equalTo(TypeDescriptor.of(Void.class)));
   }
+
+  @Test
+  public void testStructuralValueSharesSameObject() {
+assertEquals(TEST_CODER.structuralValue(null), 
TEST_CODER.structuralValue(null));
+// This is a minor performance optimization to not encode and compare 
empty byte
+// arrays.
+assertSame(TEST_CODER.structuralValue(null), 
TEST_CODER.structuralValue(null));
+  }
 }

[jira] [Resolved] (BEAM-1487) BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-1487.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in 
> precommit
> --
>
> Key: BEAM-1487
> URL: https://issues.apache.org/jira/browse/BEAM-1487
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Blocker
>  Labels: portability
> Fix For: 2.3.0
>
>
> BufferingStreamObserverTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7401/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7248/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7267/
> BeamFnLoggingClientTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7274/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7256/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7265/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7288/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7254/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7111/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-3016) BeamFnLoggingClient shutdown stability

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3016.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> BeamFnLoggingClient shutdown stability
> --
>
> Key: BEAM-3016
> URL: https://issues.apache.org/jira/browse/BEAM-3016
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
> Fix For: 2.3.0
>
>
> The BeamFnLoggingClient has a race condition where it may not shutdown 
> appropriately if the channel has an error.
> This occurs because the stream observer created by the StreamObserverFactory 
> may block indefinitely if the DirectStreamObserver is used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4002: Cherrypick of #3995 to 2.2.0: Add custom tempLocati...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4002


---

[2/2] beam git commit: This closes #4002: Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO

[jkff]

This closes #4002: Cherrypick of #3995 to 2.2.0: Add custom tempLocation 
support to BigQueryIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5601f0ae
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5601f0ae
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5601f0ae

Branch: refs/heads/release-2.2.0
Commit: 5601f0ae315e4d1bb6791ea4370df02ce7223146
Parents: 0a4774d cfbacaa
Author: Eugene Kirpichov 
Authored: Mon Oct 16 16:55:18 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 16 16:55:18 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java| 25 --
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 49 
 2 files changed, 61 insertions(+), 13 deletions(-)
--

[1/2] beam git commit: Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 0a4774d88 -> 5601f0ae3


Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cfbacaad
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cfbacaad
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cfbacaad

Branch: refs/heads/release-2.2.0
Commit: cfbacaad15867cde231836328dd73c339ff9056e
Parents: 0a4774d
Author: Yunqing Zhou 
Authored: Fri Oct 13 15:52:18 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 16 15:21:18 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java| 25 --
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 49 
 2 files changed, 61 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cfbacaad/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 6d832e4..1ccd5d6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -41,6 +41,7 @@ import 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles.Result;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
@@ -127,11 +128,13 @@ class BatchLoads
   private long maxFileSize;
   private int numFileShards;
   private Duration triggeringFrequency;
+  private ValueProvider customGcsTempLocation;
 
   BatchLoads(WriteDisposition writeDisposition, CreateDisposition 
createDisposition,
  boolean singletonTable,
  DynamicDestinations dynamicDestinations,
- Coder destinationCoder) {
+ Coder destinationCoder,
+ ValueProvider customGcsTempLocation) {
 bigQueryServices = new BigQueryServicesImpl();
 this.writeDisposition = writeDisposition;
 this.createDisposition = createDisposition;
@@ -142,6 +145,7 @@ class BatchLoads
 this.maxFileSize = DEFAULT_MAX_FILE_SIZE;
 this.numFileShards = DEFAULT_NUM_FILE_SHARDS;
 this.triggeringFrequency = null;
+this.customGcsTempLocation = customGcsTempLocation;
   }
 
   void setTestServices(BigQueryServices bigQueryServices) {
@@ -174,7 +178,16 @@ class BatchLoads
   @Override
   public void validate(PipelineOptions options) {
 // We will use a BigQuery load job -- validate the temp location.
-String tempLocation = options.getTempLocation();
+String tempLocation;
+if (customGcsTempLocation == null) {
+  tempLocation = options.getTempLocation();
+} else {
+  if (!customGcsTempLocation.isAccessible()) {
+// Can't perform verification in this case.
+return;
+  }
+  tempLocation = customGcsTempLocation.get();
+}
 checkArgument(
 !Strings.isNullOrEmpty(tempLocation),
 "BigQueryIO.Write needs a GCS temp location to store temp files.");
@@ -359,9 +372,15 @@ class BatchLoads
 new DoFn() {
   @ProcessElement
   public void getTempFilePrefix(ProcessContext c) {
+String tempLocationRoot;
+if (customGcsTempLocation != null) {
+  tempLocationRoot = customGcsTempLocation.get();
+} else {
+  tempLocationRoot = 
c.getPipelineOptions().getTempLocation();
+}
 String tempLocation =
 resolveTempLocation(
-c.getPipelineOptions().getTempLocation(),
+tempLocationRoot,
 "BigQueryWriteTemp",
 c.element());
 LOG.info(

http://git-wip-us.apache.org/repos/asf/beam/blob/cfbacaad/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/goo

[jira] [Commented] (BEAM-1487) BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206777#comment-16206777
 ] 

ASF GitHub Bot commented on BEAM-1487:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3992


> BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in 
> precommit
> --
>
> Key: BEAM-1487
> URL: https://issues.apache.org/jira/browse/BEAM-1487
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Blocker
>  Labels: portability
>
> BufferingStreamObserverTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7401/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7248/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7267/
> BeamFnLoggingClientTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7274/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7256/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7265/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7288/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7254/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7111/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3992: [BEAM-1487, BEAM-3016] Address termination correctn...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3992


---

[1/2] beam git commit: [BEAM-1487, BEAM-3016] Address termination correctness issues in BufferingStreamObserver & BeamFnLoggingClient

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master c8f3a9183 -> 3633c40b9


[BEAM-1487, BEAM-3016] Address termination correctness issues in 
BufferingStreamObserver & BeamFnLoggingClient

The issue with BeamFnLoggingClient is that we can't arriveAndDeregister during 
termination since
the onReadyHandler may also arrive at the same time which is why we swap to 
using forced termination.
Also, I added code that would guarantee that log messages produced by the 
thread which is shutting
down are guaranteed to make it (this was being caught occassionally by the 
testLogging test).

The BufferingStreamObserver was incorrectly shutting down since it may attempt 
to enqueue something
into a full queue with a reading thread that has already exitted for some 
reason so it would loop
forever attempting to insert the poison pill.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6cdea08e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6cdea08e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6cdea08e

Branch: refs/heads/master
Commit: 6cdea08e06b52bb6a34f41dc6521bcfe8f6f83cb
Parents: c8f3a91
Author: Luke Cwik 
Authored: Thu Oct 12 16:26:29 2017 -0700
Committer: Luke Cwik 
Committed: Mon Oct 16 16:31:59 2017 -0700

--
 sdks/java/harness/pom.xml   |  7 ---
 .../fn/harness/logging/BeamFnLoggingClient.java | 22 +---
 .../harness/stream/BufferingStreamObserver.java | 16 +++---
 .../stream/BufferingStreamObserverTest.java |  4 ++--
 4 files changed, 25 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6cdea08e/sdks/java/harness/pom.xml
--
diff --git a/sdks/java/harness/pom.xml b/sdks/java/harness/pom.xml
index de24f7f..e965923 100644
--- a/sdks/java/harness/pom.xml
+++ b/sdks/java/harness/pom.xml
@@ -112,13 +112,6 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-
-  
-
-
org.apache.beam.fn.harness.logging.BeamFnLoggingClientTest
-
org.apache.beam.fn.harness.stream.BufferingStreamObserverTest
-  
-
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/6cdea08e/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
--
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
index d43ab25..b19277a 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
@@ -145,12 +145,6 @@ public class BeamFnLoggingClient implements AutoCloseable {
   @Override
   public void close() throws Exception {
 try {
-  // Hang up with the server
-  logRecordHandler.close();
-
-  // Wait for the server to hang up
-  inboundObserverCompletion.get();
-} finally {
   // Reset the logging configuration to what it is at startup
   for (Logger logger : configuredLoggers) {
 logger.setLevel(null);
@@ -158,6 +152,12 @@ public class BeamFnLoggingClient implements AutoCloseable {
   configuredLoggers.clear();
   LogManager.getLogManager().readConfiguration();
 
+  // Hang up with the server
+  logRecordHandler.close();
+
+  // Wait for the server to hang up
+  inboundObserverCompletion.get();
+} finally {
   // Shut the channel down
   channel.shutdown();
   if (!channel.awaitTermination(10, TimeUnit.SECONDS)) {
@@ -255,6 +255,14 @@ public class BeamFnLoggingClient implements AutoCloseable {
   outboundObserver.onNext(builder.build());
   additionalLogEntries.clear();
 }
+
+// Perform one more final check to see if there are any log entries to 
guarantee that
+// if a log entry was added on the thread performing termination that 
we will send it.
+bufferedLogEntries.drainTo(additionalLogEntries);
+if (!additionalLogEntries.isEmpty()) {
+  outboundObserver.onNext(
+  
BeamFnApi.LogEntry.List.newBuilder().addAllLogEntries(additionalLogEntries).build());
+}
   } catch (Throwable t) {
 thrown = t;
   }
@@ -281,7 +289,7 @@ public class BeamFnLoggingClient implements AutoCloseable {
 
   // Terminate the phaser that we block on when attempting to honor flow 
control on the
   // outbound observer.
-  phaser.arriveAndDeregister();
+  phaser.forceTermination();
 
   try {
 bufferedLogWriter.get

[2/2] beam git commit: [BEAM-1487, BEAM-3016] Address termination correctness issues in BufferingStreamObserver & BeamFnLoggingClient

[lcwik]

[BEAM-1487, BEAM-3016] Address termination correctness issues in 
BufferingStreamObserver & BeamFnLoggingClient

This closes #3992


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3633c40b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3633c40b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3633c40b

Branch: refs/heads/master
Commit: 3633c40b9f75e69734484feb139c2e755ac36cd2
Parents: c8f3a91 6cdea08
Author: Luke Cwik 
Authored: Mon Oct 16 16:32:46 2017 -0700
Committer: Luke Cwik 
Committed: Mon Oct 16 16:32:46 2017 -0700

--
 sdks/java/harness/pom.xml   |  7 ---
 .../fn/harness/logging/BeamFnLoggingClient.java | 22 +---
 .../harness/stream/BufferingStreamObserver.java | 16 +++---
 .../stream/BufferingStreamObserverTest.java |  4 ++--
 4 files changed, 25 insertions(+), 24 deletions(-)
--

[jira] [Commented] (BEAM-2878) Release process for container images

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206754#comment-16206754
 ] 

ASF GitHub Bot commented on BEAM-2878:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3981


> Release process for container images
> 
>
> Key: BEAM-2878
> URL: https://issues.apache.org/jira/browse/BEAM-2878
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Kenneth Knowles
>  Labels: portability
>
> We need to push container images -- notably SDK harness container images -- 
> to a public place with each release, so that they can be referenced and used. 
> For example, https://hub.docker.com/r/apache/ might be an option, although 
> Docker hub seems more geared towards containerizing container-unware content 
> post-release.
> For users who wish to build dev (or custom) container images any location is 
> fine (personal account on docker hub, GCR, etc), but we do not want it to be 
> a requirement for normal use of official Beam releases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3981: [BEAM-2878] Use bintray as default for container im...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3981


---

[2/3] beam git commit: Add license to new file

[kenn]

Add license to new file


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f48f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f48f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f48f

Branch: refs/heads/master
Commit: f48fb5748515f5303209c631faabc7958908
Parents: ee2408b
Author: Henning Rohde 
Authored: Wed Oct 11 15:33:46 2017 -0700
Committer: Henning Rohde 
Committed: Wed Oct 11 15:33:46 2017 -0700

--
 sdks/CONTAINERS.md | 19 +++
 1 file changed, 19 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f48f/sdks/CONTAINERS.md
--
diff --git a/sdks/CONTAINERS.md b/sdks/CONTAINERS.md
index 3ee27d8..2df8901 100644
--- a/sdks/CONTAINERS.md
+++ b/sdks/CONTAINERS.md
@@ -1,3 +1,22 @@
+
+
 # Docker containers
 
 The Beam [portability effort](https://s.apache.org/beam-fn-api) aims to make 
it possible

[1/3] beam git commit: [BEAM-2878] Use bintray as default container images

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 52863efd7 -> c8f3a9183


[BEAM-2878] Use bintray as default container images


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ee2408b9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ee2408b9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ee2408b9

Branch: refs/heads/master
Commit: ee2408b9b0637d2876f246ac1acb7d358d5ddc71
Parents: 92615ca
Author: Henning Rohde 
Authored: Wed Oct 11 14:52:28 2017 -0700
Committer: Henning Rohde 
Committed: Wed Oct 11 14:52:28 2017 -0700

--
 pom.xml|   6 +-
 sdks/CONTAINERS.md | 143 
 2 files changed, 145 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ee2408b9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index efedb1d..2472d28 100644
--- a/pom.xml
+++ b/pom.xml
@@ -181,8 +181,8 @@
 
 ${user.home}/.kube/config
 
-
-${user.name}
+
+
${user.name}-docker-apache.bintray.io/beam
   
 
   pom
@@ -383,10 +383,8 @@
   default
   
 build
-push
   
   
-latest
 true
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ee2408b9/sdks/CONTAINERS.md
--
diff --git a/sdks/CONTAINERS.md b/sdks/CONTAINERS.md
new file mode 100644
index 000..3ee27d8
--- /dev/null
+++ b/sdks/CONTAINERS.md
@@ -0,0 +1,143 @@
+# Docker containers
+
+The Beam [portability effort](https://s.apache.org/beam-fn-api) aims to make 
it possible
+for any SDK to work with any runner. One aspect of the effort is the isolation 
of the SDK
+and user code execution environment from the runner execution environment using
+[docker](https://www.docker.com/), as defined in the portability
+[container contract](https://s.apache.org/beam-fn-api-container-contract).
+
+This document describes how to build and push container images to that end. 
The push
+step generally requires an account with a public docker registry, such
+as [bintray.io](bintray.io) or
+[Google Container Registry](https://cloud.google.com/container-registry). These
+instructions assume familiarity with docker and a bintray account under the
+current username with a docker repository named "apache".
+
+## How to build container images
+
+**Prerequisites**: install [docker](https://www.docker.com/) on your
+platform. You can verify that it works by running `docker images` or any other
+docker command.
+
+Run Maven with the `build-containers` profile:
+
+```
+$ pwd
+[...]/beam
+$ mvn clean install -DskipTests -Pbuild-containers
+[...]
+[INFO] --- dockerfile-maven-plugin:1.3.5:build (default) @ 
beam-sdks-python-container ---
+[INFO] Using Google application default credentials
+[INFO] loaded credentials for user account with 
clientId=[...].apps.googleusercontent.com
+[INFO] Building Docker context 
/Users/herohde/go/src/github.com/apache/beam/sdks/python/container
+[INFO] 
+[INFO] Image will be built as 
herohde-docker-apache.bintray.io/beam/python:latest
+[INFO] 
+[INFO] Step 1/4 : FROM python:2
+[INFO] Pulling from library/python
+[INFO] Digest: 
sha256:181ee8edfd9d44323c82dcba0b187f1ee2eb3d4a11c8398fc06952ed5f9ef32c
+[INFO] Status: Image is up to date for python:2
+[INFO]  ---> b1d5c2d7dda8
+[INFO] Step 2/4 : MAINTAINER "Apache Beam "
+[INFO]  ---> Running in f1bc3c4943b3
+[INFO]  ---> 9867b512e47e
+[INFO] Removing intermediate container f1bc3c4943b3
+[INFO] Step 3/4 : ADD target/linux_amd64/boot /opt/apache/beam/
+[INFO]  ---> 5cb81c3d2d90
+[INFO] Removing intermediate container 4a41ad80005a
+[INFO] Step 4/4 : ENTRYPOINT /opt/apache/beam/boot
+[INFO]  ---> Running in 40f5b945afe7
+[INFO]  ---> c8bf712741c8
+[INFO] Removing intermediate container 40f5b945afe7
+[INFO] Successfully built c8bf712741c8
+[INFO] Successfully tagged herohde-docker-apache.bintray.io/beam/python:latest
+[INFO] 
+[INFO] Detected build of image with id c8bf712741c8
+[INFO] Building jar: 
/Users/herohde/go/src/github.com/apache/beam/sdks/python/container/target/beam-sdks-python-container-2.3.0-SNAPSHOT-docker-info.jar
+[INFO] Successfully built herohde-docker-apache.bintray.io/beam/python:latest
+[INFO]
+[...]
+```
+
+Note that the container images include built content, including the Go boot
+code, so you should build from the top level directory unless you're familiar
+with Maven.
+
+**(Optional)** When built, you can see, inspect and run them locally:
+
+```
+$ docker images
+REPOSITORY   TAGIMAGE 
IDCREATED SIZE

[3/3] beam git commit: This closes #3981: [BEAM-2878] Use bintray as default for container images

[kenn]

This closes #3981: [BEAM-2878] Use bintray as default for container images

  Add license to new file
  [BEAM-2878] Use bintray as default container images


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c8f3a918
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c8f3a918
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c8f3a918

Branch: refs/heads/master
Commit: c8f3a9183aa360cbd2b7e8132791a9fde82f11b7
Parents: 52863ef f48
Author: Kenneth Knowles 
Authored: Mon Oct 16 16:12:26 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 16:12:26 2017 -0700

--
 pom.xml|   6 +-
 sdks/CONTAINERS.md | 162 
 2 files changed, 164 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c8f3a918/pom.xml
--

[3/5] beam git commit: Allow checking timers set in TriggerStateMachineTester

[kenn]

Allow checking timers set in TriggerStateMachineTester


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/64fb19da
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/64fb19da
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/64fb19da

Branch: refs/heads/master
Commit: 64fb19da42e3d26d0a9dae41b19dd7bf77ff49c7
Parents: e940456
Author: Kenneth Knowles 
Authored: Mon Oct 16 15:16:07 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 15:19:14 2017 -0700

--
 .../beam/runners/core/triggers/TriggerStateMachineTester.java  | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/64fb19da/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
index 9a10f53..b41977d 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
@@ -198,6 +198,12 @@ public class TriggerStateMachineTester {
 }
   }
 
+  /** Retrieves the next timer for this time domain, if any, for use in 
assertions. */
+  @Nullable
+  public Instant getNextTimer(TimeDomain domain) {
+return timerInternals.getNextTimer(domain);
+  }
+
   /**
* Returns {@code true} if the {@link TriggerStateMachine} under test is 
finished for the given
* window.

[1/5] beam git commit: Return null when timer not found instead of crashing

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master a5cbd764b -> 52863efd7


Return null when timer not found instead of crashing


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e940456b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e940456b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e940456b

Branch: refs/heads/master
Commit: e940456bd95da3c8b79eb4666ad09280dccaedcf
Parents: 655227a
Author: Kenneth Knowles 
Authored: Mon Oct 16 15:13:26 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 15:17:32 2017 -0700

--
 .../runners/core/InMemoryTimerInternals.java| 28 ++--
 1 file changed, 14 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e940456b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
index c29ea19..c7b4ac6 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -25,6 +25,7 @@ import com.google.common.base.MoreObjects;
 import com.google.common.collect.HashBasedTable;
 import com.google.common.collect.Table;
 import java.util.NavigableSet;
+import java.util.NoSuchElementException;
 import java.util.TreeSet;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.state.TimeDomain;
@@ -71,21 +72,20 @@ public class InMemoryTimerInternals implements 
TimerInternals {
*/
   @Nullable
   public Instant getNextTimer(TimeDomain domain) {
-final TimerData data;
-switch (domain) {
-  case EVENT_TIME:
-data = watermarkTimers.first();
-break;
-  case PROCESSING_TIME:
-data = processingTimers.first();
-break;
-  case SYNCHRONIZED_PROCESSING_TIME:
-data = synchronizedProcessingTimers.first();
-break;
-  default:
-throw new IllegalArgumentException("Unexpected time domain: " + 
domain);
+try {
+  switch (domain) {
+case EVENT_TIME:
+  return watermarkTimers.first().getTimestamp();
+case PROCESSING_TIME:
+  return processingTimers.first().getTimestamp();
+case SYNCHRONIZED_PROCESSING_TIME:
+  return synchronizedProcessingTimers.first().getTimestamp();
+default:
+  throw new IllegalArgumentException("Unexpected time domain: " + 
domain);
+  }
+} catch (NoSuchElementException exc) {
+  return null;
 }
-return (data == null) ? null : data.getTimestamp();
   }
 
   private NavigableSet timersForDomain(TimeDomain domain) {

[GitHub] beam pull request #4001: [BEAM-3052] Set end-of-window timer in AfterWaterma...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4001


---

[4/5] beam git commit: Do not set EOW timer in TriggerStateMachineTester

[kenn]

Do not set EOW timer in TriggerStateMachineTester


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3d36f63c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3d36f63c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3d36f63c

Branch: refs/heads/master
Commit: 3d36f63cbb6de5ce7115d34a1432835407cd666a
Parents: 45c65c5
Author: Kenneth Knowles 
Authored: Mon Oct 16 15:03:33 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 15:19:14 2017 -0700

--
 .../beam/runners/core/triggers/TriggerStateMachineTester.java | 7 ---
 1 file changed, 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3d36f63c/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
index b41977d..0f38be0 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
@@ -269,11 +269,6 @@ public class TriggerStateMachineTester {
 
 for (W window : assignedWindows) {
   activeWindows.addActiveForTesting(window);
-
-  // Today, triggers assume onTimer firing at the watermark time, 
whether or not they
-  // explicitly set the timer themselves. So this tester must set it.
-  timerInternals.setTimer(
-  TimerData.of(windowNamespace(window), window.maxTimestamp(), 
TimeDomain.EVENT_TIME));
 }
 
 windowedValues.add(WindowedValue.of(value, timestamp, assignedWindows, 
PaneInfo.NO_FIRING));
@@ -357,8 +352,6 @@ public class TriggerStateMachineTester {
 
executableTrigger.invokeOnMerge(contextFactory.createOnMergeContext(mergeResult,
 new TestTimers(windowNamespace(mergeResult)), executableTrigger,
 getFinishedSet(mergeResult), mergingFinishedSets));
-timerInternals.setTimer(TimerData.of(
-windowNamespace(mergeResult), mergeResult.maxTimestamp(), 
TimeDomain.EVENT_TIME));
   }
 });
   }

[jira] [Commented] (BEAM-3052) ReduceFnRunner sets end-of-window hold even when no data is buffered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206734#comment-16206734
 ] 

ASF GitHub Bot commented on BEAM-3052:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4001


> ReduceFnRunner sets end-of-window hold even when no data is buffered
> 
>
> Key: BEAM-3052
> URL: https://issues.apache.org/jira/browse/BEAM-3052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> If you set a trigger that ignores the end of the window (like repeated 
> processing time trigger) and an early firing emits all the data, the 
> ReduceFnRunner will leave an end-of-window watermark hold in place but only a 
> GC timer.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[5/5] beam git commit: This closes #4001: [BEAM-3052] Set end-of-window timer in AfterWatermarkStateMachine

[kenn]

This closes #4001: [BEAM-3052] Set end-of-window timer in 
AfterWatermarkStateMachine

  Do not set EOW timer in TriggerStateMachineTester
  Set end-of-window timer in AfterWatermarkStateMachine
  Allow checking timers set in TriggerStateMachineTester
  Return null when timer not found instead of crashing


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/52863efd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/52863efd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/52863efd

Branch: refs/heads/master
Commit: 52863efd7478d406597b5b162c61701314155f25
Parents: a5cbd76 3d36f63
Author: Kenneth Knowles 
Authored: Mon Oct 16 15:20:27 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 15:20:27 2017 -0700

--
 .../runners/core/InMemoryTimerInternals.java| 28 +--
 .../triggers/AfterWatermarkStateMachine.java|  8 +-
 .../AfterWatermarkStateMachineTest.java | 29 
 .../triggers/TriggerStateMachineTester.java | 13 -
 4 files changed, 56 insertions(+), 22 deletions(-)
--

[2/5] beam git commit: Set end-of-window timer in AfterWatermarkStateMachine

[kenn]

Set end-of-window timer in AfterWatermarkStateMachine


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/45c65c55
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/45c65c55
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/45c65c55

Branch: refs/heads/master
Commit: 45c65c557bf79f8b4c5975eee15b5763e45882be
Parents: 64fb19d
Author: Kenneth Knowles 
Authored: Thu Oct 12 13:38:27 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 16 15:19:14 2017 -0700

--
 .../triggers/AfterWatermarkStateMachine.java|  8 +-
 .../AfterWatermarkStateMachineTest.java | 29 
 2 files changed, 36 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/45c65c55/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
index 509c96b..2c99722 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
@@ -100,6 +100,10 @@ public class AfterWatermarkStateMachine {
 
 @Override
 public void onElement(OnElementContext c) throws Exception {
+  if (!endOfWindowReached(c)) {
+c.setTimer(c.window().maxTimestamp(), TimeDomain.EVENT_TIME);
+  }
+
   if (!c.trigger().isMerging()) {
 // If merges can never happen, we just run the unfinished subtrigger
 c.trigger().firstUnfinishedSubTrigger().invokeOnElement(c);
@@ -270,7 +274,9 @@ public class AfterWatermarkStateMachine {
   // We're interested in knowing when the input watermark passes the end 
of the window.
   // (It is possible this has already happened, in which case the timer 
will be fired
   // almost immediately).
-  c.setTimer(c.window().maxTimestamp(), TimeDomain.EVENT_TIME);
+  if (!endOfWindowReached(c)) {
+c.setTimer(c.window().maxTimestamp(), TimeDomain.EVENT_TIME);
+  }
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/45c65c55/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
index 45a5cfb..65c8be3 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
@@ -17,8 +17,11 @@
  */
 package org.apache.beam.runners.core.triggers;
 
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.Mockito.doNothing;
 import static org.mockito.Mockito.verify;
@@ -26,6 +29,7 @@ import static org.mockito.Mockito.when;
 
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnMergeContext;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachineTester.SimpleTriggerStateMachineTester;
+import org.apache.beam.sdk.state.TimeDomain;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.Sessions;
@@ -104,6 +108,31 @@ public class AfterWatermarkStateMachineTest {
   }
 
   @Test
+  public void testTimerForEndOfWindow() throws Exception {
+tester = TriggerStateMachineTester.forTrigger(
+AfterWatermarkStateMachine.pastEndOfWindow(),
+FixedWindows.of(Duration.millis(100)));
+
+assertThat(tester.getNextTimer(TimeDomain.EVENT_TIME), nullValue());
+injectElements(1);
+IntervalWindow window = new IntervalWindow(new Instant(0), new 
Instant(100));
+assertThat(tester.getNextTimer(TimeDomain.EVENT_TIME), 
equalTo(window.maxTimestamp()));
+  }
+
+  @Test
+  public void testTimerForEndOfWindowCompound() throws Exception {
+tester =
+TriggerStateMachineTester.forTrigger(
+
AfterWatermarkStateMachine.pastEndOfWindow().withEarlyFirings(NeverStateMachine.ever()),
+FixedWindo

[GitHub] beam pull request #4002: Cherrypick of #3995 to 2.2.0: Add custom tempLocati...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/4002

Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO

#3995 

R: @reuvenlax 
CC: @azurezyq 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam cp-3995

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4002


commit cfbacaad15867cde231836328dd73c339ff9056e
Author: Yunqing Zhou 
Date:   2017-10-13T22:52:18Z

Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO.




---

[jira] [Commented] (BEAM-3059) Timeout option for individual integration tests

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206677#comment-16206677
 ] 

Mark Liu commented on BEAM-3059:


Some explanation: we want to have a timeout option for each individual 
integration test, so that test will fail if it exceeds the time limit. 

This will have the following benefits:
1. build fail fast if one test stall or running too long. 
2. improve debugging information in the situation when Jenkins build stall and 
timeout.

> Timeout option for individual integration tests
> ---
>
> Key: BEAM-3059
> URL: https://issues.apache.org/jira/browse/BEAM-3059
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> cc: [~markflyhigh]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3995: Add custom gcs temp location support to BigQueryIO.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3995


---

[1/2] beam git commit: Add custom tempLocation support to BigQueryIO.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master ec052bb44 -> a5cbd764b


Add custom tempLocation support to BigQueryIO.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ec58a80c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ec58a80c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ec58a80c

Branch: refs/heads/master
Commit: ec58a80ca0f913c85d5f17cba3535243cd010876
Parents: ec052bb
Author: Yunqing Zhou 
Authored: Fri Oct 13 15:52:18 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 16 15:17:59 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java| 25 --
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 49 
 2 files changed, 61 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ec58a80c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 6d832e4..1ccd5d6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -41,6 +41,7 @@ import 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles.Result;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
@@ -127,11 +128,13 @@ class BatchLoads
   private long maxFileSize;
   private int numFileShards;
   private Duration triggeringFrequency;
+  private ValueProvider customGcsTempLocation;
 
   BatchLoads(WriteDisposition writeDisposition, CreateDisposition 
createDisposition,
  boolean singletonTable,
  DynamicDestinations dynamicDestinations,
- Coder destinationCoder) {
+ Coder destinationCoder,
+ ValueProvider customGcsTempLocation) {
 bigQueryServices = new BigQueryServicesImpl();
 this.writeDisposition = writeDisposition;
 this.createDisposition = createDisposition;
@@ -142,6 +145,7 @@ class BatchLoads
 this.maxFileSize = DEFAULT_MAX_FILE_SIZE;
 this.numFileShards = DEFAULT_NUM_FILE_SHARDS;
 this.triggeringFrequency = null;
+this.customGcsTempLocation = customGcsTempLocation;
   }
 
   void setTestServices(BigQueryServices bigQueryServices) {
@@ -174,7 +178,16 @@ class BatchLoads
   @Override
   public void validate(PipelineOptions options) {
 // We will use a BigQuery load job -- validate the temp location.
-String tempLocation = options.getTempLocation();
+String tempLocation;
+if (customGcsTempLocation == null) {
+  tempLocation = options.getTempLocation();
+} else {
+  if (!customGcsTempLocation.isAccessible()) {
+// Can't perform verification in this case.
+return;
+  }
+  tempLocation = customGcsTempLocation.get();
+}
 checkArgument(
 !Strings.isNullOrEmpty(tempLocation),
 "BigQueryIO.Write needs a GCS temp location to store temp files.");
@@ -359,9 +372,15 @@ class BatchLoads
 new DoFn() {
   @ProcessElement
   public void getTempFilePrefix(ProcessContext c) {
+String tempLocationRoot;
+if (customGcsTempLocation != null) {
+  tempLocationRoot = customGcsTempLocation.get();
+} else {
+  tempLocationRoot = 
c.getPipelineOptions().getTempLocation();
+}
 String tempLocation =
 resolveTempLocation(
-c.getPipelineOptions().getTempLocation(),
+tempLocationRoot,
 "BigQueryWriteTemp",
 c.element());
 LOG.info(

http://git-wip-us.apache.org/repos/asf/beam/blob/ec58a80c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/

[2/2] beam git commit: This closes #3995: Add custom gcs temp location support to BigQueryIO

[jkff]

This closes #3995: Add custom gcs temp location support to BigQueryIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a5cbd764
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a5cbd764
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a5cbd764

Branch: refs/heads/master
Commit: a5cbd764bb81ca2aefe225d62e15e222a2266f0b
Parents: ec052bb ec58a80
Author: Eugene Kirpichov 
Authored: Mon Oct 16 15:18:31 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 16 15:18:31 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java| 25 --
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 49 
 2 files changed, 61 insertions(+), 13 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4175

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3052) ReduceFnRunner sets end-of-window hold even when no data is buffered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206653#comment-16206653
 ] 

ASF GitHub Bot commented on BEAM-3052:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4001

[BEAM-3052] Set end-of-window timer in AfterWatermarkStateMachine

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

The state machines for the variants of `AfterWatermark.pastEndOfWindow()` 
were inconsistent in when they set a timer or not. Since this state machine 
should be decoupled from whatever context it is being run in, it should set a 
timer for the time it is interested in.

This is deliberately peeled off #3988 since it is simpler and more 
definitely correct. The other bits in that PR are under further revision.

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam AfterWatermark-timer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4001


commit db17a188ab98379e2985d01e63eaa2c4bb5893fd
Author: Kenneth Knowles 
Date:   2017-10-12T20:38:27Z

Set end-of-window timer in AfterWatermarkStateMachine




> ReduceFnRunner sets end-of-window hold even when no data is buffered
> 
>
> Key: BEAM-3052
> URL: https://issues.apache.org/jira/browse/BEAM-3052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> If you set a trigger that ignores the end of the window (like repeated 
> processing time trigger) and an early firing emits all the data, the 
> ReduceFnRunner will leave an end-of-window watermark hold in place but only a 
> GC timer.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4001: [BEAM-3052] Set end-of-window timer in AfterWaterma...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4001

[BEAM-3052] Set end-of-window timer in AfterWatermarkStateMachine

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

The state machines for the variants of `AfterWatermark.pastEndOfWindow()` 
were inconsistent in when they set a timer or not. Since this state machine 
should be decoupled from whatever context it is being run in, it should set a 
timer for the time it is interested in.

This is deliberately peeled off #3988 since it is simpler and more 
definitely correct. The other bits in that PR are under further revision.

R: @bjchambers 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam AfterWatermark-timer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4001


commit db17a188ab98379e2985d01e63eaa2c4bb5893fd
Author: Kenneth Knowles 
Date:   2017-10-12T20:38:27Z

Set end-of-window timer in AfterWatermarkStateMachine




---

[jira] [Commented] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206651#comment-16206651
 ] 

ASF GitHub Bot commented on BEAM-3063:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4000

[BEAM-3063] Improve VoidCoder structural value to use a single shared 
instance.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam3063

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4000


commit 0d8626e5a0c99420535375a2162e22e60eeb28df
Author: Luke Cwik 
Date:   2017-10-16T20:56:04Z

[BEAM-3063] Improve VoidCoder structural value to use a single shared 
instance.




> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> VoidCoder structuralValue always performs an encode on something which is 
> effectively zero bytes instead of returning a trivial object which is always 
> equals to itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4000: [BEAM-3063] Improve VoidCoder structural value to u...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4000

[BEAM-3063] Improve VoidCoder structural value to use a single shared 
instance.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam3063

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4000


commit 0d8626e5a0c99420535375a2162e22e60eeb28df
Author: Luke Cwik 
Date:   2017-10-16T20:56:04Z

[BEAM-3063] Improve VoidCoder structural value to use a single shared 
instance.




---

[jira] [Updated] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3063:

Description: VoidCoder structuralValue always performs an encode on 
something which is effectively zero bytes instead of returning a trivial object 
which is always equals to itself.  (was: Currently Coder.java says that 
structuralValue(x).equals(@code structuralValue(y))

> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> VoidCoder structuralValue always performs an encode on something which is 
> effectively zero bytes instead of returning a trivial object which is always 
> equals to itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2682) Merge AvroIOTest and AvroIOTransformTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206642#comment-16206642
 ] 

ASF GitHub Bot commented on BEAM-2682:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3999

[BEAM-2682] Deletes AvroIOTransformTest

Instead, merges the little test coverage it provided into AvroIOTest.

This supersedes https://github.com/apache/beam/pull/3948 - merging these 
two test classes is long overdue and AvroIOTransformTest was quite bad code:

* the parameterized tests in AvroIOTransformTest were considerably more 
verbose and less readable than doing the same thing without parameterization
* it provided little additional coverage over AvroIOTest
* it didn't use the "write pipeline, then read pipeline" recommended 
approach

R: @echauchot 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam rm-avroio-transform

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3999.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3999


commit b59a7423fbdad9dd104a0c3161378ff343cf4401
Author: Eugene Kirpichov 
Date:   2017-10-16T21:48:17Z

[BEAM-2682] Deletes AvroIOTransformTest

Instead, merges the little test coverage it provided into AvroIOTest.




> Merge AvroIOTest and AvroIOTransformTest
> 
>
> Key: BEAM-2682
> URL: https://issues.apache.org/jira/browse/BEAM-2682
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> These two tests seem to have exactly the same purpose. They should be merged 
> into AvroIOTest.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3999: [BEAM-2682] Deletes AvroIOTransformTest

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3999

[BEAM-2682] Deletes AvroIOTransformTest

Instead, merges the little test coverage it provided into AvroIOTest.

This supersedes https://github.com/apache/beam/pull/3948 - merging these 
two test classes is long overdue and AvroIOTransformTest was quite bad code:

* the parameterized tests in AvroIOTransformTest were considerably more 
verbose and less readable than doing the same thing without parameterization
* it provided little additional coverage over AvroIOTest
* it didn't use the "write pipeline, then read pipeline" recommended 
approach

R: @echauchot 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam rm-avroio-transform

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3999.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3999


commit b59a7423fbdad9dd104a0c3161378ff343cf4401
Author: Eugene Kirpichov 
Date:   2017-10-16T21:48:17Z

[BEAM-2682] Deletes AvroIOTransformTest

Instead, merges the little test coverage it provided into AvroIOTest.




---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3302

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3064) Update dataflow runner containers for the release bracn

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3064:
-

 Summary: Update dataflow runner containers for the release bracn
 Key: BEAM-3064
 URL: https://issues.apache.org/jira/browse/BEAM-3064
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Affects Versions: 2.2.0
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Critical


Blocked by:
https://github.com/apache/beam/pull/3970
https://github.com/apache/beam/pull/3941 - cp into release branch.

cc: [~reuvenlax] [~robertwb] [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3029) BigTable integration tests failing on Dataflow: UserAgent must not be empty

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206593#comment-16206593
 ] 

ASF GitHub Bot commented on BEAM-3029:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3998

[BEAM-3029] Sets user agent in BigTableIO.Read.getBigTableService().

Cherry-picking this commit to 2.2.0 release branch.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam 
bigtable_read_it_fix_cerrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3998.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3998


commit 25cab6be8d763a03d2a37f0647698cba79df6ac5
Author: chamik...@google.com 
Date:   2017-10-16T07:50:03Z

Sets user agent in BigTableIO.Read.getBigTableService().




> BigTable integration tests failing on Dataflow: UserAgent must not be empty
> ---
>
> Key: BEAM-3029
> URL: https://issues.apache.org/jira/browse/BEAM-3029
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.2.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.bigtable/BigtableReadIT/testE2EBigtableRead/
> {code}
> java.lang.IllegalArgumentException: UserAgent must not be empty or null
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:77)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:351)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4086

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3998: [BEAM-3029] Sets user agent in BigTableIO.Read.getB...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3998

[BEAM-3029] Sets user agent in BigTableIO.Read.getBigTableService().

Cherry-picking this commit to 2.2.0 release branch.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam 
bigtable_read_it_fix_cerrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3998.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3998


commit 25cab6be8d763a03d2a37f0647698cba79df6ac5
Author: chamik...@google.com 
Date:   2017-10-16T07:50:03Z

Sets user agent in BigTableIO.Read.getBigTableService().




---

[jira] [Updated] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3063:

Issue Type: Improvement  (was: Bug)

> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> Currently Coder.java says that structuralValue(x).equals(@code 
> structuralValue(y)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3063:

Description: Currently Coder.java says that structuralValue(x).equals(@code 
structuralValue(y)

> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> Currently Coder.java says that structuralValue(x).equals(@code 
> structuralValue(y)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3063:

Issue Type: Bug  (was: Improvement)

> VoidCoder should implement structuralValue instead of forcing encoding of 
> zero bytes
> 
>
> Key: BEAM-3063
> URL: https://issues.apache.org/jira/browse/BEAM-3063
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3063) VoidCoder should implement structuralValue instead of forcing encoding of zero bytes

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3063:
---

 Summary: VoidCoder should implement structuralValue instead of 
forcing encoding of zero bytes
 Key: BEAM-3063
 URL: https://issues.apache.org/jira/browse/BEAM-3063
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3049) Java SDK Harness bundles non-relocated code, including Dataflow runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206575#comment-16206575
 ] 

Kenneth Knowles edited comment on BEAM-3049 at 10/16/17 8:55 PM:
-

Finally tracked it down. It is 
https://github.com/apache/beam/commit/c18f15cdf7eeabcaf64a2c808372683c3d823d4d#diff-d6ed52a12f15794e45d1eca0cbc47380


was (Author: kenn):
Ah, I didn't track it down, but it is 
https://github.com/apache/beam/commit/c18f15cdf7eeabcaf64a2c808372683c3d823d4d#diff-d6ed52a12f15794e45d1eca0cbc47380

> Java SDK Harness bundles non-relocated code, including Dataflow runner
> --
>
> Key: BEAM-3049
> URL: https://issues.apache.org/jira/browse/BEAM-3049
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This causes a problem if something depends on the harness but does not want 
> the harness's copy of its dependencies. I know we intend to break the 
> dependency on the Dataflow runner. It also bundles a couple other things 
> unshaded.
> Mostly, the harness should be executed entirely containerized so it doesn't 
> matter, in which case there's no need to relocate anything, and bundling is 
> just a convenience. But we should have a clear policy that we adhere to. 
> Either it is a library and should have good hygeine, or if it doesn't have 
> good hygeine it must not be used as a library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3049) Java SDK Harness bundles non-relocated code, including Dataflow runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3049:
-

Assignee: Kenneth Knowles  (was: Thomas Groh)

> Java SDK Harness bundles non-relocated code, including Dataflow runner
> --
>
> Key: BEAM-3049
> URL: https://issues.apache.org/jira/browse/BEAM-3049
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This causes a problem if something depends on the harness but does not want 
> the harness's copy of its dependencies. I know we intend to break the 
> dependency on the Dataflow runner. It also bundles a couple other things 
> unshaded.
> Mostly, the harness should be executed entirely containerized so it doesn't 
> matter, in which case there's no need to relocate anything, and bundling is 
> just a convenience. But we should have a clear policy that we adhere to. 
> Either it is a library and should have good hygeine, or if it doesn't have 
> good hygeine it must not be used as a library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3049) Java SDK Harness bundles non-relocated code, including Dataflow runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206575#comment-16206575
 ] 

Kenneth Knowles commented on BEAM-3049:
---

Ah, I didn't track it down, but it is 
https://github.com/apache/beam/commit/c18f15cdf7eeabcaf64a2c808372683c3d823d4d#diff-d6ed52a12f15794e45d1eca0cbc47380

> Java SDK Harness bundles non-relocated code, including Dataflow runner
> --
>
> Key: BEAM-3049
> URL: https://issues.apache.org/jira/browse/BEAM-3049
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This causes a problem if something depends on the harness but does not want 
> the harness's copy of its dependencies. I know we intend to break the 
> dependency on the Dataflow runner. It also bundles a couple other things 
> unshaded.
> Mostly, the harness should be executed entirely containerized so it doesn't 
> matter, in which case there's no need to relocate anything, and bundling is 
> just a convenience. But we should have a clear policy that we adhere to. 
> Either it is a library and should have good hygeine, or if it doesn't have 
> good hygeine it must not be used as a library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3029) BigTable integration tests failing on Dataflow: UserAgent must not be empty

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206536#comment-16206536
 ] 

ASF GitHub Bot commented on BEAM-3029:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3996


> BigTable integration tests failing on Dataflow: UserAgent must not be empty
> ---
>
> Key: BEAM-3029
> URL: https://issues.apache.org/jira/browse/BEAM-3029
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.2.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.bigtable/BigtableReadIT/testE2EBigtableRead/
> {code}
> java.lang.IllegalArgumentException: UserAgent must not be empty or null
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:77)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:351)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3996: [BEAM-3029] Sets user agent in BigTableIO.Read.getB...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3996


---

[1/2] beam git commit: Sets user agent in BigTableIO.Read.getBigTableService().

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master 245d77338 -> ec052bb44


Sets user agent in BigTableIO.Read.getBigTableService().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6b25948b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6b25948b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6b25948b

Branch: refs/heads/master
Commit: 6b25948b4f56e4d45e1f9e03eb19c6077413a80e
Parents: 245d773
Author: chamik...@google.com 
Authored: Mon Oct 16 00:50:03 2017 -0700
Committer: chamik...@google.com 
Committed: Mon Oct 16 13:14:17 2017 -0700

--
 .../main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java   | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6b25948b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 47efa08..ba6c0b6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -415,6 +415,7 @@ public class BigtableIO {
 return getBigtableService();
   }
   BigtableOptions.Builder clonedOptions = getBigtableOptions().toBuilder();
+  clonedOptions.setUserAgent(pipelineOptions.getUserAgent());
   if (getBigtableOptions().getCredentialOptions()
   .getCredentialType() == CredentialType.DefaultCredentials) {
 clonedOptions.setCredentialOptions(

[2/2] beam git commit: This closes #3996

[chamikara]

This closes #3996


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ec052bb4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ec052bb4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ec052bb4

Branch: refs/heads/master
Commit: ec052bb44ce40ae5fda2fbbd94fbc2b97ac363f5
Parents: 245d773 6b25948
Author: chamik...@google.com 
Authored: Mon Oct 16 13:14:25 2017 -0700
Committer: chamik...@google.com 
Committed: Mon Oct 16 13:14:25 2017 -0700

--
 .../main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java   | 1 +
 1 file changed, 1 insertion(+)
--

[jira] [Commented] (BEAM-2822) Add support for progress reporting in fn API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206530#comment-16206530
 ] 

ASF GitHub Bot commented on BEAM-2822:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3939


> Add support for progress reporting in fn API
> 
>
> Key: BEAM-2822
> URL: https://issues.apache.org/jira/browse/BEAM-2822
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Vikas Kedigehalli
>Assignee: Robert Bradshaw
>Priority: Minor
>  Labels: portability
>
> https://s.apache.org/beam-fn-api-progress-reporting
> Note that the ULR reference implementation, when ready, should be useful for 
> every runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3939: [BEAM-2822] Add progress metric reporting to Python...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3939


---

[1/2] beam git commit: Closes #3939

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 014614b69 -> 245d77338


Closes #3939


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/245d7733
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/245d7733
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/245d7733

Branch: refs/heads/master
Commit: 245d77338be8677df7ea9cbd75b1b8ab8c4da831
Parents: 014614b 3ad8479
Author: Robert Bradshaw 
Authored: Mon Oct 16 13:09:31 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 16 13:09:31 2017 -0700

--
 .../src/main/proto/beam_fn_api.proto| 10 ++-
 .../runners/portability/fn_api_runner.py| 15 +++-
 .../runners/portability/fn_api_runner_test.py   | 67 +++
 .../runners/worker/bundle_processor.py  | 90 ++--
 .../apache_beam/runners/worker/opcounters.py|  9 +-
 .../apache_beam/runners/worker/operations.pxd   |  3 +-
 .../apache_beam/runners/worker/operations.py| 55 
 .../apache_beam/runners/worker/sdk_worker.py| 22 +++--
 .../apache_beam/runners/worker/statesampler.pyx |  3 +
 .../runners/worker/statesampler_fake.py | 15 
 10 files changed, 226 insertions(+), 63 deletions(-)
--

[2/2] beam git commit: Add progress metrics to Python SDK.

[robertwb]

Add progress metrics to Python SDK.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3ad84791
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3ad84791
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3ad84791

Branch: refs/heads/master
Commit: 3ad84791d4d85896f46b7956b5bd8045cdc4a0e9
Parents: 014614b
Author: Robert Bradshaw 
Authored: Mon Oct 2 17:20:38 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 16 13:09:31 2017 -0700

--
 .../src/main/proto/beam_fn_api.proto| 10 ++-
 .../runners/portability/fn_api_runner.py| 15 +++-
 .../runners/portability/fn_api_runner_test.py   | 67 +++
 .../runners/worker/bundle_processor.py  | 90 ++--
 .../apache_beam/runners/worker/opcounters.py|  9 +-
 .../apache_beam/runners/worker/operations.pxd   |  3 +-
 .../apache_beam/runners/worker/operations.py| 55 
 .../apache_beam/runners/worker/sdk_worker.py| 22 +++--
 .../apache_beam/runners/worker/statesampler.pyx |  3 +
 .../runners/worker/statesampler_fake.py | 15 
 10 files changed, 226 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3ad84791/model/fn-execution/src/main/proto/beam_fn_api.proto
--
diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto 
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index 7d3e05b..132d366 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -216,8 +216,10 @@ message Metrics {
   message PTransform {
 // Metrics that are measured for processed and active element groups.
 message Measured {
-  // (Required) Map from local input name to number of elements processed
+  // (Optional) Map from local input name to number of elements processed
   // from this input.
+  // If unset, assumed to be the sum of the outputs of all producers to
+  // this transform (for ProcessedElements) and 0 (for ActiveElements).
   map input_element_counts = 1;
 
   // (Required) Map from local output name to number of elements produced
@@ -225,8 +227,8 @@ message Metrics {
   map output_element_counts = 2;
 
   // (Optional) The total time spent so far in processing the elements in
-  // this group.
-  int64 total_time_spent = 3;
+  // this group, in seconds.
+  double total_time_spent = 3;
 
   // TODO: Add other element group level metrics.
 }
@@ -542,7 +544,7 @@ message StateKey {
 string ptransform_id = 1;
 // (Required) The id of the user state.
 string user_state_id = 2;
-// (Required) The window encoded in a nested context. 
+// (Required) The window encoded in a nested context.
 bytes window = 3;
 // (Required) The key of the currently executing element encoded in a
 // nested context.

http://git-wip-us.apache.org/repos/asf/beam/blob/3ad84791/sdks/python/apache_beam/runners/portability/fn_api_runner.py
--
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 20a4a61..463f78f 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -632,16 +632,18 @@ class 
FnApiRunner(maptask_executor_runner.MapTaskExecutorRunner):
   controller = FnApiRunner.GrpcController(self._sdk_harness_factory)
 else:
   controller = FnApiRunner.DirectController()
+metrics_by_stage = {}
 
 try:
   pcoll_buffers = collections.defaultdict(list)
   for stage in stages:
-self.run_stage(
-controller, pipeline_components, stage, pcoll_buffers, safe_coders)
+metrics_by_stage[stage.name] = self.run_stage(
+controller, pipeline_components, stage,
+pcoll_buffers, safe_coders).process_bundle.metrics
 finally:
   controller.close()
 
-return maptask_executor_runner.WorkerRunnerResult(PipelineState.DONE)
+return RunnerResult(PipelineState.DONE, metrics_by_stage)
 
   def run_stage(
   self, controller, pipeline_components, stage, pcoll_buffers, 
safe_coders):
@@ -757,6 +759,7 @@ class 
FnApiRunner(maptask_executor_runner.MapTaskExecutorRunner):
   # These should be the only two identifiers we produce for now,
   # but special side input writes may go here.
   raise NotImplementedError(pcoll_id)
+return result
 
   # This is the "old" way of executing pipelines.
   # TODO(robertwb): Remove once runner API supports side inputs.
@@ -1071,6 +1074,12 @@ class 
FnApiRunner(maptask_executor_

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5039

[Apache Jenkins Server]

See