[jira] [Commented] (BEAM-2316) GcsUtil matches GCS glob * as though it is **

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015148#comment-16015148
 ] 

Kenneth Knowles commented on BEAM-2316:
---

[~dhalp...@google.com] you may have some stake in the appropriate behavior 
here, or the best path forwards (rollback vs rollforwards vs define the prior 
behavior to have been a bug)

> GcsUtil matches GCS glob * as though it is **
> -
>
> Key: BEAM-2316
> URL: https://issues.apache.org/jira/browse/BEAM-2316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Assignee: Devon Meunier
>
> A pattern like {{"prefix/*"}} previously did not match files in 
> subdirectories, but was restricted by the {{/}} character. Now it matches 
> things like {{"prefix/subdir/foo.txt"}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2316) GcsUtil matches GCS glob * as though it is **

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2316:
--
Summary: GcsUtil matches GCS glob * as though it is **  (was: GCS path 
matching)

> GcsUtil matches GCS glob * as though it is **
> -
>
> Key: BEAM-2316
> URL: https://issues.apache.org/jira/browse/BEAM-2316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Assignee: Devon Meunier
>
> A pattern like {{"prefix/*"}} previously did not match files in 
> subdirectories, but was restricted by the {{/}} character. Now it matches 
> things like {{"prefix/subdir/foo.txt"}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2316) GCS path matching

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2316:
-

 Summary: GCS path matching
 Key: BEAM-2316
 URL: https://issues.apache.org/jira/browse/BEAM-2316
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Kenneth Knowles
Assignee: Devon Meunier


A pattern like {{"prefix/*"}} previously did not match files in subdirectories, 
but was restricted by the {{/}} character. Now it matches things like 
{{"prefix/subdir/foo.txt"}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015131#comment-16015131
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3172

[BEAM-79] Sync gearpump-runner with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3172


commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
Author: Dan Halperin 
Date:   2017-05-06T00:16:34Z

[BEAM-2190] pom.xml: do a better job of dependency management

Even if Beam appears to have the correct dependencies, we cannot
guarantee that modules that depend on us transitively get the right
dependencies. For example, even though grpc-protobuf-lite has
protobuf-lite excluded, and the Maven Enforcer banned-dependencies
check passes... if a user happens to get a transitive dependency on
grpc-all first, they may pull in grpc-protobuf from that other source
without the exclusion. Thus we need to exclude protobuf-lite from
grpc-all as well.

While we're here, also add guava-jdk5 to the set of banned dependencies,
though (as above) we cannot currently properly identify the places it
might be transitively exposed in a users' pom.xml.

commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
Author: Dan Halperin 
Date:   2017-05-08T18:35:56Z

This closes #2934

commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
Author: Dan Halperin 
Date:   2017-05-06T02:06:03Z

[BEAM-2212] FileBasedSource: improve message when logging.

ValueProvider should not be printed, rather the string instead.

commit 9423babd8f827e843723c218441e9a91aaa7b361
Author: Dan Halperin 
Date:   2017-05-08T16:59:16Z

[BEAM-2212] FileBasedSource: refactor to remove uses of 
fileOrPatternSpec.get()

Makes it less likely to have errors from printing ValueProviders instead of 
runtime values

commit e1791c3f83a007013625b43f9976e7ffa64e8a49
Author: Dan Halperin 
Date:   2017-05-08T18:48:42Z

This closes #2956

commit 13e51c9cd114eec73c47b71f46214dd92ee81048
Author: Dan Halperin 
Date:   2017-05-06T02:49:29Z

Convert Executor Services to use Daemon Threads

This will cause the DirectRunner to automatically shut down when the
worker threads are shut down.

commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
Author: Thomas Groh 
Date:   2017-05-08T19:52:09Z

This closes #2954

commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
Author: Sourabh Bajaj 
Date:   2017-05-07T20:55:20Z

[BEAM-2206] Move pipelineOptions into options modules

commit 604c2f59e2a928a95106814b0b55b8097dc39faa
Author: Ahmet Altay 
Date:   2017-05-08T19:56:15Z

This closes #2925

commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
Author: Reuven Lax 
Date:   2017-05-08T16:06:55Z

TableDescription is allowed to be null.

commit c481396edc7a3e906cd79dcc2a356baf9c93895e
Author: Eugene Kirpichov 
Date:   2017-05-08T20:41:01Z

This closes #2953

commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
Author: Reuven Lax 
Date:   2017-05-07T18:50:08Z

Shortens PubsubIO names for reading/writing messages.

commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
Author: Eugene Kirpichov 
Date:   2017-05-08T20:45:25Z

This closes #2950

commit b7635aadc2532b5c3bdc38f070865091ebf901c1
Author: Ahmet Altay 
Date:   2017-05-08T20:42:02Z

Remove apex README.md. This information is already in the website in
quickstart and apex runner pages.
https://github.com/apache/beam-site/pull/232 moves small bits of missing
content.

commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
Author: Ahmet Altay 
Date:   2017-05-08T21:28:36Z

This closes #2962

commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
Author: Ahmet 

[GitHub] beam pull request #3172: [BEAM-79] Sync gearpump-runner with master

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3172

[BEAM-79] Sync gearpump-runner with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3172


commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
Author: Dan Halperin 
Date:   2017-05-06T00:16:34Z

[BEAM-2190] pom.xml: do a better job of dependency management

Even if Beam appears to have the correct dependencies, we cannot
guarantee that modules that depend on us transitively get the right
dependencies. For example, even though grpc-protobuf-lite has
protobuf-lite excluded, and the Maven Enforcer banned-dependencies
check passes... if a user happens to get a transitive dependency on
grpc-all first, they may pull in grpc-protobuf from that other source
without the exclusion. Thus we need to exclude protobuf-lite from
grpc-all as well.

While we're here, also add guava-jdk5 to the set of banned dependencies,
though (as above) we cannot currently properly identify the places it
might be transitively exposed in a users' pom.xml.

commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
Author: Dan Halperin 
Date:   2017-05-08T18:35:56Z

This closes #2934

commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
Author: Dan Halperin 
Date:   2017-05-06T02:06:03Z

[BEAM-2212] FileBasedSource: improve message when logging.

ValueProvider should not be printed, rather the string instead.

commit 9423babd8f827e843723c218441e9a91aaa7b361
Author: Dan Halperin 
Date:   2017-05-08T16:59:16Z

[BEAM-2212] FileBasedSource: refactor to remove uses of 
fileOrPatternSpec.get()

Makes it less likely to have errors from printing ValueProviders instead of 
runtime values

commit e1791c3f83a007013625b43f9976e7ffa64e8a49
Author: Dan Halperin 
Date:   2017-05-08T18:48:42Z

This closes #2956

commit 13e51c9cd114eec73c47b71f46214dd92ee81048
Author: Dan Halperin 
Date:   2017-05-06T02:49:29Z

Convert Executor Services to use Daemon Threads

This will cause the DirectRunner to automatically shut down when the
worker threads are shut down.

commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
Author: Thomas Groh 
Date:   2017-05-08T19:52:09Z

This closes #2954

commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
Author: Sourabh Bajaj 
Date:   2017-05-07T20:55:20Z

[BEAM-2206] Move pipelineOptions into options modules

commit 604c2f59e2a928a95106814b0b55b8097dc39faa
Author: Ahmet Altay 
Date:   2017-05-08T19:56:15Z

This closes #2925

commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
Author: Reuven Lax 
Date:   2017-05-08T16:06:55Z

TableDescription is allowed to be null.

commit c481396edc7a3e906cd79dcc2a356baf9c93895e
Author: Eugene Kirpichov 
Date:   2017-05-08T20:41:01Z

This closes #2953

commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
Author: Reuven Lax 
Date:   2017-05-07T18:50:08Z

Shortens PubsubIO names for reading/writing messages.

commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
Author: Eugene Kirpichov 
Date:   2017-05-08T20:45:25Z

This closes #2950

commit b7635aadc2532b5c3bdc38f070865091ebf901c1
Author: Ahmet Altay 
Date:   2017-05-08T20:42:02Z

Remove apex README.md. This information is already in the website in
quickstart and apex runner pages.
https://github.com/apache/beam-site/pull/232 moves small bits of missing
content.

commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
Author: Ahmet Altay 
Date:   2017-05-08T21:28:36Z

This closes #2962

commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
Author: Ahmet Altay 
Date:   2017-05-08T22:00:33Z

Remove readme.md files that are covered in web site

commit 152c5bcc7f6fb53565ed132653384f4244542b59
Author: Ahmet Altay 
Date:   2017-05-08T22:04:46Z

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #23

[Apache Jenkins Server]

See 


--
[...truncated 355.30 KB...]
2017-05-18T02:47:59.093 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
2017-05-18T02:47:59.102 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
 (7 KB at 742.7 KB/sec)
2017-05-18T02:47:59.107 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
2017-05-18T02:47:59.117 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
 (3 KB at 260.1 KB/sec)
2017-05-18T02:47:59.120 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-18T02:47:59.131 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 145.1 KB/sec)
2017-05-18T02:47:59.135 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-18T02:47:59.145 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 455.1 KB/sec)
2017-05-18T02:47:59.150 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-18T02:47:59.161 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 380.5 KB/sec)
2017-05-18T02:47:59.165 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-18T02:47:59.175 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 170.7 KB/sec)
2017-05-18T02:47:59.179 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-18T02:47:59.190 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 133.0 KB/sec)
2017-05-18T02:47:59.193 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-18T02:47:59.200 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 370.4 KB/sec)
2017-05-18T02:47:59.204 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-18T02:47:59.216 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 345.8 KB/sec)
2017-05-18T02:47:59.220 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-18T02:47:59.231 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 128.3 KB/sec)
2017-05-18T02:47:59.235 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-18T02:47:59.244 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 388.5 KB/sec)
2017-05-18T02:47:59.249 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-18T02:47:59.262 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 159.6 KB/sec)
2017-05-18T02:47:59.269 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-18T02:47:59.281 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 169.1 KB/sec)
2017-05-18T02:47:59.294 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
2017-05-18T02:47:59.299 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
2017-05-18T02:47:59.300 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-05-18T02:47:59.301 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
2017-05-18T02:47:59.305 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-05-18T02:47:59.314 [INFO] Downloaded: 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2853

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #22

[Apache Jenkins Server]

See 


Changes:

[tgroh] Translate a Pipeline in SdkComponents

[klk] Use stable naming strategy for ByteBuddy invokers

--
[...truncated 355.36 KB...]
2017-05-17T22:56:40.564 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
2017-05-17T22:56:40.575 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
 (7 KB at 607.7 KB/sec)
2017-05-17T22:56:40.580 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
2017-05-17T22:56:40.591 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
 (3 KB at 236.4 KB/sec)
2017-05-17T22:56:40.595 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-17T22:56:40.605 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 159.6 KB/sec)
2017-05-17T22:56:40.608 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-17T22:56:40.618 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 556.2 KB/sec)
2017-05-17T22:56:40.622 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-17T22:56:40.631 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 523.2 KB/sec)
2017-05-17T22:56:40.635 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-17T22:56:40.642 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 243.9 KB/sec)
2017-05-17T22:56:40.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-17T22:56:40.659 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 112.5 KB/sec)
2017-05-17T22:56:40.663 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-17T22:56:40.673 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 288.1 KB/sec)
2017-05-17T22:56:40.677 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-17T22:56:40.686 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 461.0 KB/sec)
2017-05-17T22:56:40.691 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-17T22:56:40.702 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 128.3 KB/sec)
2017-05-17T22:56:40.706 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-17T22:56:40.715 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 388.5 KB/sec)
2017-05-17T22:56:40.720 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-17T22:56:40.729 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 259.3 KB/sec)
2017-05-17T22:56:40.733 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-17T22:56:40.744 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 184.5 KB/sec)
2017-05-17T22:56:40.755 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-05-17T22:56:40.756 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-05-17T22:56:40.756 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
2017-05-17T22:56:40.757 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
2017-05-17T22:56:40.757 [INFO] Downloading: 

[jira] [Updated] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-981:
-
Fix Version/s: (was: 2.1.0)

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Kobi Salant
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
> at 
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
> at 
> io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:845)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:750)
> at 
> 

[GitHub] beam pull request #3170: A couple of worker fixes.

[robertwb]

GitHub user robertwb reopened a pull request:

https://github.com/apache/beam/pull/3170

A couple of worker fixes.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3170


commit 53f111c4ccb17daf1178241fd247b6cf7d7b67dd
Author: Robert Bradshaw 
Date:   2017-05-17T21:25:08Z

A couple of worker fixes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3170: A couple of worker fixes.

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/3170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2314) Add ValidatesRunner test for custom merging windows

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014818#comment-16014818
 ] 

Aljoscha Krettek commented on BEAM-2314:


[~echauchot], I thought you might be interested in doing this, based on the 
work on NEXMark.

> Add ValidatesRunner test for custom merging windows
> ---
>
> Key: BEAM-2314
> URL: https://issues.apache.org/jira/browse/BEAM-2314
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Aljoscha Krettek
>Assignee: Davor Bonaci
>
> While working on BEAM-160 it was discovered that not all Runners supported 
> custom merging {{WindowFns}}. This is now fixed but we should still add a 
> {{ValidatesRunner}} test for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2288) Refine DSL interface

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-2288.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Refine DSL interface
> 
>
> Key: BEAM-2288
> URL: https://issues.apache.org/jira/browse/BEAM-2288
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> As part of BEAM-2010, it update the interface to explain SQL query, to link 
> with Pipeline, so below part is supported:
> {code}
> //prepare environment of BeamSQL
> BeamSQLEnvironment sqlEnv = BeamSQLEnvironment.create();
> //register table metadata
> sqlEnv.addTableMetadata(String tableName, BeamSqlTable tableMetadata);
> //register UDF
> sqlEnv.registerUDF(String functionName, Method udfMethod);
> //explain a SQL statement, SELECT only, and return as a PCollection;
> PCollection phase1Stream = sqlEnv.explainSQL(pipeline, String 
> sqlStatement);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2288) Refine DSL interface

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2288.


> Refine DSL interface
> 
>
> Key: BEAM-2288
> URL: https://issues.apache.org/jira/browse/BEAM-2288
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> As part of BEAM-2010, it update the interface to explain SQL query, to link 
> with Pipeline, so below part is supported:
> {code}
> //prepare environment of BeamSQL
> BeamSQLEnvironment sqlEnv = BeamSQLEnvironment.create();
> //register table metadata
> sqlEnv.addTableMetadata(String tableName, BeamSqlTable tableMetadata);
> //register UDF
> sqlEnv.registerUDF(String functionName, Method udfMethod);
> //explain a SQL statement, SELECT only, and return as a PCollection;
> PCollection phase1Stream = sqlEnv.explainSQL(pipeline, String 
> sqlStatement);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2288) Refine DSL interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014812#comment-16014812
 ] 

ASF GitHub Bot commented on BEAM-2288:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3137


> Refine DSL interface
> 
>
> Key: BEAM-2288
> URL: https://issues.apache.org/jira/browse/BEAM-2288
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> As part of BEAM-2010, it update the interface to explain SQL query, to link 
> with Pipeline, so below part is supported:
> {code}
> //prepare environment of BeamSQL
> BeamSQLEnvironment sqlEnv = BeamSQLEnvironment.create();
> //register table metadata
> sqlEnv.addTableMetadata(String tableName, BeamSqlTable tableMetadata);
> //register UDF
> sqlEnv.registerUDF(String functionName, Method udfMethod);
> //explain a SQL statement, SELECT only, and return as a PCollection;
> PCollection phase1Stream = sqlEnv.explainSQL(pipeline, String 
> sqlStatement);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3137: [BEAM-2288] Refine DSL interface

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3171: [BEAM-2164] Fix generate-sources.sh if directory co...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3171

[BEAM-2164] Fix generate-sources.sh if directory contains space

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


[generate-sources.sh](https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh)
 is executed in beam-sdks-java-maven-archetypes-examples and 
beam-sdks-java-maven-archetypes-examples-java8. If project directory contains 
space, mvn build of the two module will fail because the space character is not 
handled properly.

Test is done by successfully running mvn clean build. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
fix-generate-sources

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3171


commit e8bc4f994e10b0d70f3e1123904360d5fb45c3b3
Author: Mark Liu 
Date:   2017-05-17T21:24:25Z

[BEAM-2164] Fix generate-sources.sh if directory contains space




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2164) Build beam-sdks-java-maven-archetypes-examples Failed If Dir Contains Space

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014809#comment-16014809
 ] 

ASF GitHub Bot commented on BEAM-2164:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3171

[BEAM-2164] Fix generate-sources.sh if directory contains space

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


[generate-sources.sh](https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh)
 is executed in beam-sdks-java-maven-archetypes-examples and 
beam-sdks-java-maven-archetypes-examples-java8. If project directory contains 
space, mvn build of the two module will fail because the space character is not 
handled properly.

Test is done by successfully running mvn clean build. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
fix-generate-sources

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3171


commit e8bc4f994e10b0d70f3e1123904360d5fb45c3b3
Author: Mark Liu 
Date:   2017-05-17T21:24:25Z

[BEAM-2164] Fix generate-sources.sh if directory contains space




> Build beam-sdks-java-maven-archetypes-examples Failed If Dir Contains Space
> ---
>
> Key: BEAM-2164
> URL: https://issues.apache.org/jira/browse/BEAM-2164
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Error log from [Jenkins console 
> log|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=JDK%201.8%20(latest),label=beam/6/console]:
> {code}
> rsync: change_dir 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk\#012.\#012(latest)/label/beam/sdks/java/maven-archetypes/examples/../../../../examples/java/src/main/java/org/apache/beam/examples"
>  failed: No such file or directory (2)
> rsync error: some files/attrs were not transferred (see previous errors) 
> (code 23) at main.c(1183) [sender=3.1.0]
> 2017-05-03\T\19:20:56.905 [ERROR] Command execution failed.
> {code}
> beam-sdks-java-maven-archetypes-examples execute a [bash 
> script|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh]
>  through 
> [exec-maven-plugin|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/pom.xml#L79].
>  Looks like it fails to create a directory which contains space in [previous 
> step|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh#L32].
>  The root directory of above build is 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.8 (latest)/label/beam/...".
> Similar error also happens in 
> [JDK1.7|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=JDK%201.7%20(latest),label=beam/8/]
>  and 
> [OpenJDK7|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=OpenJDK%207%20(on%20Ubuntu%20only),label=beam/8/].
> I simply run 
> [generate-sources.sh|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh]
>  from my Linux machine in JDK1.8 under contain-space directory and it works 
> perfectly. I guess exec-maven-plugin works differently when handling bash 
> commands that contains space, but I failed to solve this problem by modifying 
> commands in script.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3170: A couple of worker fixes.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3170

A couple of worker fixes.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3170


commit 53f111c4ccb17daf1178241fd247b6cf7d7b67dd
Author: Robert Bradshaw 
Date:   2017-05-17T21:25:08Z

A couple of worker fixes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-2288] This closes #3137

[jbonofre]

[BEAM-2288] This closes #3137


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/25babc99
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/25babc99
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/25babc99

Branch: refs/heads/DSL_SQL
Commit: 25babc999e964775834503e25043e3a8d7b6d030
Parents: f96f9f6 7b3f7a2
Author: Jean-Baptiste Onofré 
Authored: Wed May 17 23:13:24 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed May 17 23:13:24 2017 +0200

--
 .../beam/dsls/sql/BeamSQLEnvironment.java   | 142 +++
 .../beam/dsls/sql/example/BeamSqlExample.java   |  23 ++-
 .../dsls/sql/planner/BeamPipelineCreator.java   |   9 +-
 .../beam/dsls/sql/planner/BeamQueryPlanner.java |  34 ++---
 .../beam/dsls/sql/planner/BeamSqlRunner.java| 103 --
 .../beam/dsls/sql/planner/BasePlanner.java  |   9 +-
 .../sql/planner/BeamGroupByExplainTest.java |  14 +-
 .../sql/planner/BeamGroupByPipelineTest.java|  19 +--
 .../sql/planner/BeamInvalidGroupByTest.java |  10 +-
 .../BeamPlannerAggregationSubmitTest.java   |  17 ++-
 .../sql/planner/BeamPlannerExplainTest.java |   7 +-
 .../dsls/sql/planner/BeamPlannerSubmitTest.java |  11 +-
 .../beam/dsls/sql/rel/BeamSortRelTest.java  |  40 --
 13 files changed, 251 insertions(+), 187 deletions(-)
--

[1/2] beam git commit: [BEAM-2288] Refine DSL interface as design doc of BEAM-2010: 1. rename BeamSqlRunner to BeamSQLEnvironment; 2. Move pipeline as external parameter, to integrate with none-DSL co

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL f96f9f680 -> 25babc999


[BEAM-2288] Refine DSL interface as design doc of BEAM-2010:
1. rename BeamSqlRunner to BeamSQLEnvironment;
2. Move pipeline as external parameter, to integrate with none-DSL components;


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7b3f7a2b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7b3f7a2b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7b3f7a2b

Branch: refs/heads/DSL_SQL
Commit: 7b3f7a2b5bb5017819e8c8c53620d3cf97a38008
Parents: f96f9f6
Author: mingmxu 
Authored: Sun May 14 11:02:37 2017 -0700
Committer: Jean-Baptiste Onofré 
Committed: Wed May 17 21:03:13 2017 +0200

--
 .../beam/dsls/sql/BeamSQLEnvironment.java   | 142 +++
 .../beam/dsls/sql/example/BeamSqlExample.java   |  23 ++-
 .../dsls/sql/planner/BeamPipelineCreator.java   |   9 +-
 .../beam/dsls/sql/planner/BeamQueryPlanner.java |  34 ++---
 .../beam/dsls/sql/planner/BeamSqlRunner.java| 103 --
 .../beam/dsls/sql/planner/BasePlanner.java  |   9 +-
 .../sql/planner/BeamGroupByExplainTest.java |  14 +-
 .../sql/planner/BeamGroupByPipelineTest.java|  19 +--
 .../sql/planner/BeamInvalidGroupByTest.java |  10 +-
 .../BeamPlannerAggregationSubmitTest.java   |  17 ++-
 .../sql/planner/BeamPlannerExplainTest.java |   7 +-
 .../dsls/sql/planner/BeamPlannerSubmitTest.java |  11 +-
 .../beam/dsls/sql/rel/BeamSortRelTest.java  |  40 --
 13 files changed, 251 insertions(+), 187 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7b3f7a2b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/BeamSQLEnvironment.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/BeamSQLEnvironment.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/BeamSQLEnvironment.java
new file mode 100644
index 000..cdb25f5
--- /dev/null
+++ b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/BeamSQLEnvironment.java
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.dsls.sql;
+
+import java.io.Serializable;
+import org.apache.beam.dsls.sql.planner.BeamQueryPlanner;
+import org.apache.beam.dsls.sql.rel.BeamRelNode;
+import org.apache.beam.dsls.sql.schema.BaseBeamTable;
+import org.apache.beam.dsls.sql.schema.BeamSQLRow;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.schema.Schema;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.impl.ScalarFunctionImpl;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.RelConversionException;
+import org.apache.calcite.tools.ValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code BeamSQLEnvironment} is the integrated environment of BeamSQL.
+ * It provides runtime context to execute SQL queries as Beam pipeline,
+ * including table metadata, SQL engine and a Beam pipeline translator.
+ *
+ * 1. BeamSQL as DSL
+ * BeamSQL as DSL enables developers to embed SQL queries when 
writing a Beam pipeline.
+ * A typical pipeline with BeamSQL DSL is:
+ * 
+ *{@code
+PipelineOptions options =  PipelineOptionsFactory...
+Pipeline pipeline = Pipeline.create(options);
+
+//prepare environment of BeamSQL
+BeamSQLEnvironment sqlEnv = BeamSQLEnvironment.create();
+//register table metadata
+sqlEnv.addTableMetadata(String tableName, BeamSqlTable tableMetadata);
+//register UDF
+sqlEnv.registerUDF(String functionName, Method udfMethod);
+
+
+//explain a SQL statement, SELECT only, and return as a PCollection;
+PCollection phase1Stream = sqlEnv.explainSQL(pipeline, String 
sqlStatement);
+//A PCollection explained by BeamSQL can be converted into a table, and apply 
queries on it;

[jira] [Assigned] (BEAM-2164) Build beam-sdks-java-maven-archetypes-examples Failed If Dir Contains Space

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu reassigned BEAM-2164:
--

Assignee: Mark Liu  (was: Jean-Baptiste Onofré)

> Build beam-sdks-java-maven-archetypes-examples Failed If Dir Contains Space
> ---
>
> Key: BEAM-2164
> URL: https://issues.apache.org/jira/browse/BEAM-2164
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Error log from [Jenkins console 
> log|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=JDK%201.8%20(latest),label=beam/6/console]:
> {code}
> rsync: change_dir 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk\#012.\#012(latest)/label/beam/sdks/java/maven-archetypes/examples/../../../../examples/java/src/main/java/org/apache/beam/examples"
>  failed: No such file or directory (2)
> rsync error: some files/attrs were not transferred (see previous errors) 
> (code 23) at main.c(1183) [sender=3.1.0]
> 2017-05-03\T\19:20:56.905 [ERROR] Command execution failed.
> {code}
> beam-sdks-java-maven-archetypes-examples execute a [bash 
> script|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh]
>  through 
> [exec-maven-plugin|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/pom.xml#L79].
>  Looks like it fails to create a directory which contains space in [previous 
> step|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh#L32].
>  The root directory of above build is 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.8 (latest)/label/beam/...".
> Similar error also happens in 
> [JDK1.7|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=JDK%201.7%20(latest),label=beam/8/]
>  and 
> [OpenJDK7|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/jdk=OpenJDK%207%20(on%20Ubuntu%20only),label=beam/8/].
> I simply run 
> [generate-sources.sh|https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/generate-sources.sh]
>  from my Linux machine in JDK1.8 under contain-space directory and it works 
> perfectly. I guess exec-maven-plugin works differently when handling bash 
> commands that contains space, but I failed to solve this problem by modifying 
> commands in script.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1553) OutputTimeFn maybe should be allowed to move elements back in time

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014772#comment-16014772
 ] 

Kenneth Knowles commented on BEAM-1553:
---

I think just removing the crash is fine. Let it be moved.

> OutputTimeFn maybe should be allowed to move elements back in time
> --
>
> Key: BEAM-1553
> URL: https://issues.apache.org/jira/browse/BEAM-1553
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Minor
>  Labels: starter
>
> EG in this flink failure
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Flink/org.apache.beam$beam-runners-flink_2.10/1749/testReport/junit/org.apache.beam.sdk.transforms.windowing/WindowTest/testNoWindowFnDoesNotReassignWindows/
> Caused by: java.lang.IllegalStateException: OutputTimeFn moved element from 
> 1970-01-01T00:00:00.015Z to earlier time 1970-01-01T00:00:00.009Z for window 
> [1970-01-01T00:00:00.005Z..1970-01-01T00:00:00.010Z)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1553) OutputTimeFn maybe should be allowed to move elements back in time

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1553:
--
Labels: starter  (was: )

> OutputTimeFn maybe should be allowed to move elements back in time
> --
>
> Key: BEAM-1553
> URL: https://issues.apache.org/jira/browse/BEAM-1553
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: starter
>
> EG in this flink failure
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Flink/org.apache.beam$beam-runners-flink_2.10/1749/testReport/junit/org.apache.beam.sdk.transforms.windowing/WindowTest/testNoWindowFnDoesNotReassignWindows/
> Caused by: java.lang.IllegalStateException: OutputTimeFn moved element from 
> 1970-01-01T00:00:00.015Z to earlier time 1970-01-01T00:00:00.009Z for window 
> [1970-01-01T00:00:00.005Z..1970-01-01T00:00:00.010Z)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1553) OutputTimeFn maybe should be allowed to move elements back in time

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1553:
-

Assignee: (was: Kenneth Knowles)

> OutputTimeFn maybe should be allowed to move elements back in time
> --
>
> Key: BEAM-1553
> URL: https://issues.apache.org/jira/browse/BEAM-1553
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Minor
>  Labels: starter
>
> EG in this flink failure
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Flink/org.apache.beam$beam-runners-flink_2.10/1749/testReport/junit/org.apache.beam.sdk.transforms.windowing/WindowTest/testNoWindowFnDoesNotReassignWindows/
> Caused by: java.lang.IllegalStateException: OutputTimeFn moved element from 
> 1970-01-01T00:00:00.015Z to earlier time 1970-01-01T00:00:00.009Z for window 
> [1970-01-01T00:00:00.005Z..1970-01-01T00:00:00.010Z)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3154

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2315) Add ValidatesRunner test for non-blocking job submission

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-2315:
--

 Summary: Add ValidatesRunner test for non-blocking job submission
 Key: BEAM-2315
 URL: https://issues.apache.org/jira/browse/BEAM-2315
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


At the very least the Flink Runner does not support non-blocking job submission 
and {{PipelineResult.cancel()}}/{{.waitUntilFinish()}}. (BEAM-593)

The PR for BEAM-593 contains a test for this that is specific to the Flink 
Runner. This can easily be implemented as a {{ValidatesRunner}} test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1663) Generated package names should be deterministic and consistent across runs.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-1663.
---
   Resolution: Fixed
Fix Version/s: 2.1.0

> Generated package names should be deterministic and consistent across runs.
> ---
>
> Key: BEAM-1663
> URL: https://issues.apache.org/jira/browse/BEAM-1663
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-java-core
>Reporter: Alex Amato
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>
> This makes it difficult for error/stack trace processing systems to identify 
> that these two errors are the same.
> These two errors contain generated package names with random strings in them, 
> would it be possible to make the generated package names consistent across 
> each job run? From what I understand the random string is to prevent two 
> package names from colliding. Perhaps we could have the name be of the 
> generated packages be based off the functions they are wrapping/referring to?
> YZjse2Tt
> 1TK4aKgJ
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$1TK4aKgJ.invokeProcessElement
>  (Unknown Source)
> ---
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$YZjse2Tt.invokeProcessElement
>  (Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2314) Add ValidatesRunner test for custom merging windows

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-2314:
--

 Summary: Add ValidatesRunner test for custom merging windows
 Key: BEAM-2314
 URL: https://issues.apache.org/jira/browse/BEAM-2314
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Aljoscha Krettek
Assignee: Davor Bonaci


While working on BEAM-160 it was discovered that not all Runners supported 
custom merging {{WindowFns}}. This is now fixed but we should still add a 
{{ValidatesRunner}} test for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2852

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1663) Generated package names should be deterministic and consistent across runs.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014657#comment-16014657
 ] 

ASF GitHub Bot commented on BEAM-1663:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3151


> Generated package names should be deterministic and consistent across runs.
> ---
>
> Key: BEAM-1663
> URL: https://issues.apache.org/jira/browse/BEAM-1663
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-java-core
>Reporter: Alex Amato
>Assignee: Kenneth Knowles
>
> This makes it difficult for error/stack trace processing systems to identify 
> that these two errors are the same.
> These two errors contain generated package names with random strings in them, 
> would it be possible to make the generated package names consistent across 
> each job run? From what I understand the random string is to prevent two 
> package names from colliding. Perhaps we could have the name be of the 
> generated packages be based off the functions they are wrapping/referring to?
> YZjse2Tt
> 1TK4aKgJ
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$1TK4aKgJ.invokeProcessElement
>  (Unknown Source)
> ---
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$YZjse2Tt.invokeProcessElement
>  (Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3151: [BEAM-1663] Use stable naming strategy for ByteBudd...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3151


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3151: Use stable naming strategy for ByteBuddy invokers

[kenn]

This closes #3151: Use stable naming strategy for ByteBuddy invokers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5e3c5c65
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5e3c5c65
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5e3c5c65

Branch: refs/heads/master
Commit: 5e3c5c6574bc70320683d6c16fc3b11791a77418
Parents: 467e68f 6c45ebf
Author: Kenneth Knowles 
Authored: Wed May 17 12:26:41 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 17 12:26:41 2017 -0700

--
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 10 ++--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 20 
 .../reflect/StableInvokerNamingStrategy.java| 54 
 .../transforms/reflect/DoFnInvokersTest.java| 19 +++
 .../transforms/reflect/OnTimerInvokersTest.java | 32 
 5 files changed, 118 insertions(+), 17 deletions(-)
--

[1/2] beam git commit: Use stable naming strategy for ByteBuddy invokers

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 467e68f93 -> 5e3c5c657


Use stable naming strategy for ByteBuddy invokers

This helps to coalesce related failures across multiple
JVM instances, over time, across machines, etc.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6c45ebfb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6c45ebfb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6c45ebfb

Branch: refs/heads/master
Commit: 6c45ebfb2832caae99f4992920adbb7d19dce005
Parents: d6cc850
Author: Kenneth Knowles 
Authored: Wed Mar 8 17:02:05 2017 -0800
Committer: Kenneth Knowles 
Committed: Wed May 17 10:40:00 2017 -0700

--
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 10 ++--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 20 
 .../reflect/StableInvokerNamingStrategy.java| 54 
 .../transforms/reflect/DoFnInvokersTest.java| 19 +++
 .../transforms/reflect/OnTimerInvokersTest.java | 32 
 5 files changed, 118 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6c45ebfb/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
index 8ae2c65..5d5887a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
@@ -29,7 +29,6 @@ import java.util.List;
 import java.util.Map;
 import javax.annotation.Nullable;
 import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.NamingStrategy;
 import net.bytebuddy.description.field.FieldDescription;
 import net.bytebuddy.description.method.MethodDescription;
 import net.bytebuddy.description.modifier.Visibility;
@@ -282,12 +281,9 @@ public class ByteBuddyDoFnInvokerFactory implements 
DoFnInvokerFactory {
 // Create subclasses inside the target class, to have access to
 // private and package-private bits
 .with(
-new NamingStrategy.SuffixingRandom("auxiliary") {
-  @Override
-  public String subclass(TypeDescription.Generic superClass) {
-return super.name(clazzDescription);
-  }
-})
+StableInvokerNamingStrategy.forDoFnClass(fnClass)
+.withSuffix(DoFnInvoker.class.getSimpleName()))
+
 // class  extends DoFnInvokerBase {
 .subclass(DoFnInvokerBase.class, 
ConstructorStrategy.Default.NO_CONSTRUCTORS)
 

http://git-wip-us.apache.org/repos/asf/beam/blob/6c45ebfb/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
index 123808c..e031337 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
@@ -21,12 +21,12 @@ import com.google.common.base.CharMatcher;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
 import com.google.common.cache.LoadingCache;
+import com.google.common.io.BaseEncoding;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
 import java.util.ArrayList;
 import java.util.concurrent.ExecutionException;
 import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.NamingStrategy;
 import net.bytebuddy.description.method.MethodDescription;
 import net.bytebuddy.description.modifier.FieldManifestation;
 import net.bytebuddy.description.modifier.Visibility;
@@ -150,20 +150,20 @@ class ByteBuddyOnTimerInvokerFactory implements 
OnTimerInvokerFactory {
 
 final TypeDescription clazzDescription = new 
TypeDescription.ForLoadedType(fnClass);
 
-final String className =
-"auxiliary_OnTimer_" + 
CharMatcher.JAVA_LETTER_OR_DIGIT.retainFrom(timerId);
+final String suffix =
+String.format(
+"%s$%s$%s",
+OnTimerInvoker.class.getSimpleName(),
+CharMatcher.javaLetterOrDigit().retainFrom(timerId),
+   

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3153

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2313) Continuously execute CassandraIOIT

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014641#comment-16014641
 ] 

Jean-Baptiste Onofré commented on BEAM-2313:


[~jasonkuster] when you have a chance to take a look on  this, I'm ready to 
help. Thanks !

> Continuously execute CassandraIOIT
> --
>
> Key: BEAM-2313
> URL: https://issues.apache.org/jira/browse/BEAM-2313
> Project: Beam
>  Issue Type: Task
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jason Kuster
>
> It would be great to establishing continuously running IT coverage for 
> CassandraIO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2313) Continuously execute CassandraIOIT

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-2313:
--

Assignee: Jason Kuster  (was: Davor Bonaci)

> Continuously execute CassandraIOIT
> --
>
> Key: BEAM-2313
> URL: https://issues.apache.org/jira/browse/BEAM-2313
> Project: Beam
>  Issue Type: Task
>  Components: build-system, sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jason Kuster
>
> It would be great to establishing continuously running IT coverage for 
> CassandraIO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2313) Continuously execute CassandraIOIT

[JIRA]

Jean-Baptiste Onofré created BEAM-2313:
--

 Summary: Continuously execute CassandraIOIT
 Key: BEAM-2313
 URL: https://issues.apache.org/jira/browse/BEAM-2313
 Project: Beam
  Issue Type: Task
  Components: build-system, sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Davor Bonaci


It would be great to establishing continuously running IT coverage for 
CassandraIO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2312) Create CamelIO

[JIRA]

Jean-Baptiste Onofré created BEAM-2312:
--

 Summary: Create CamelIO
 Key: BEAM-2312
 URL: https://issues.apache.org/jira/browse/BEAM-2312
 Project: Beam
  Issue Type: Task
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


The idea is to be able to define a Camel URI in a IO and to bootstrap an 
internal CamelContext and route in a DoFn. I already started a PoC around that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014562#comment-16014562
 ] 

Kenneth Knowles commented on BEAM-981:
--

I think we should drop the fix version for work that is ongoing, and set it up 
when it gets fixed. The first stable release needed an explicit burndown, but 
going forward releases should be mostly time based.

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Kobi Salant
> Fix For: 2.1.0
>
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
> at 
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
> at 
> 

[jira] [Updated] (BEAM-2219) All TestRunners or no TestRunners should be registered

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2219:
--
Fix Version/s: (was: 2.1.0)

> All TestRunners or no TestRunners should be registered
> --
>
> Key: BEAM-2219
> URL: https://issues.apache.org/jira/browse/BEAM-2219
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Daniel Halperin
>
> Right now, {{TestXRunner}} is key to get unit tests working but is probably 
> not a user-facing runner. However, they seem to be in the {{main}} (not test) 
> artifacts [at least for Dataflow] and to be registered pipeline runners.
> We should make all the runners behave the same way w.r.t. test runner 
> registration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1107) Display user names for steps in the Flink Web UI

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1107:
--
Fix Version/s: (was: 2.1.0)

> Display user names for steps in the Flink Web UI
> 
>
> Key: BEAM-1107
> URL: https://issues.apache.org/jira/browse/BEAM-1107
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Daniel Halperin
>Assignee: Aljoscha Krettek
> Attachments: screenshot-1.png
>
>
> [copying in-person / email discussion at Strata Singapore to JIRA]
> The FlinkBatchTransformTranslators use transform.getName() [1] -- this is the 
> "SDK name" for the transform.
> The "user name" for the transform is not available here, it is in fact on the 
> TransformHierarchy.Node as node.getFullName() [2].
> getFullName() is used some in Flink, but not when setting step names.
> I drafted a quick commit that sort of propagates the user names to the web UI 
> (but only for DataSource, and still too verbose: 
> https://github.com/dhalperi/incubator-beam/commit/a2f1fb06b22a85ec738e4f2a604c9a129891916c)
> Before this change, the "ReadLines" step showed up as: "DataSource (at 
> Read(CompressedSource) 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))"
> With this change, it shows up as "DataSource (at ReadLines/Read 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))"
> which I think is closer. [I'd still like it to JUST be "ReadLines/Read" e.g.].
> Thoughts?
> [1] 
> https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java#L129
> [2] 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java#L252



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2299) Beam repo build fails in Windows OS

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2299:
--
Summary: Beam repo build fails in Windows OS  (was: Beam repo build failes 
in Windows OS)

> Beam repo build fails in Windows OS
> ---
>
> Key: BEAM-2299
> URL: https://issues.apache.org/jira/browse/BEAM-2299
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Jason Kuster
> Fix For: 2.1.0
>
>
> Need to run unit tests in Windows OS.
> Currently, many unit tests fail when doing "mvn clean install" in Windows OS.
> [ERROR] Errors:
> [ERROR]   AvroSourceTest.testCreationWithSchema:403 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   AvroSourceTest.testGetCurrentFromUnstartedReader:305 ? IllegalState 
> Un
> able to ...
> [ERROR]   AvroSourceTest.testGetProgressFromUnstartedReader:204 ? 
> IllegalState U
> nable to...
> [ERROR]   AvroSourceTest.testMultipleFiles:390 ? IllegalState Unable to find 
> reg
> istrar f...
> [ERROR]   AvroSourceTest.testProgress:225 ? IllegalState Unable to find 
> registra
> r for c
> [ERROR]   AvroSourceTest.testProgressEmptySource:278 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   AvroSourceTest.testReadMetadataWithCodecs:676 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testReadSchemaString:688 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testReadWithDifferentCodecs:158 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   AvroSourceTest.testSchemaIsInterned:460 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testSchemaStringIsInterned:441 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testSchemaUpdate:425 ? IllegalState Unable to find 
> regi
> strar fo...
> [ERROR]   AvroSourceTest.testSplitAtFraction:176 ? IllegalState Unable to 
> find r
> egistrar...
> [ERROR]   AvroSourceTest.testSplitAtFractionExhaustive:322 ? IllegalState 
> Unable
>  to find...
> [ERROR]   AvroSourceTest.testSplitsWithSmallBlocks:341 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   CompressedSourceTest.testEmptyGzipProgress:646 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   CompressedSourceTest.testGzipProgress:673 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   CompressedSourceTest.testSplittableProgress:739 ? IllegalState 
> Unable
> to find ...
> [ERROR]   CompressedSourceTest.testUncompressedFileIsSplittable:333 ? 
> IllegalSta
> te Unabl...
> [ERROR]   CompressedSourceTest.testUnsplittable:715 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   FileBasedSinkTest.testCopyToOutputFiles:301 ? IllegalState Unable 
> to f
> ind regi...
> [ERROR]   
> FileBasedSinkTest.testFinalize:154->generateTemporaryFilesForFinalize:
> 189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeMultipleCalls:161->generateTemporaryFile
> sForFinalize:189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeWithIntermediateState:171->generateTempo
> raryFilesForFinalize:189 ? IO
> [ERROR]   FileBasedSourceTest.testCloseUnstartedFilePatternReader:390 ? 
> IllegalS
> tate Una...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFile:746 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFilePattern:772 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testFractionConsumedWhenReadingFilepattern:422 
> ? I
> llegalState
> [ERROR]   FileBasedSourceTest.testFullyReadFilePattern:370 ? IllegalState 
> Unable
>  to find...
> [ERROR]   FileBasedSourceTest.testFullyReadFilePatternFirstRecordEmpty:461 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testFullyReadSingleFile:346 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfFilePattern:792 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfSingleFile:681 ? 
> IllegalState U
> nable to...
> [ERROR]   FileBasedSourceTest.testReadEverythingFromFileWithSplits:502 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadFileWithSplitsWithEmptyRange:578 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadRangeAtEnd:659 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   FileBasedSourceTest.testReadRangeAtMiddle:637 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   FileBasedSourceTest.testReadRangeAtStart:472 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddle:546 ? 
> Il
> legalState
> [ERROR]   
> FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddleOfHeader:
> 615 ? IllegalState
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromStart:517 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testSplitAtFraction:815 ? IllegalState Unable 
> 

[jira] [Assigned] (BEAM-2272) Test scope & runtime dependencies need reevaluation

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2272:
-

Assignee: (was: Thomas Groh)

> Test scope & runtime dependencies need reevaluation
> ---
>
> Key: BEAM-2272
> URL: https://issues.apache.org/jira/browse/BEAM-2272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, sdk-java-core
>Reporter: Daniel Halperin
>
> Now that SDK core has been thinned out, many of the modules that have it in 
> their {{dependenciesToScan}} will need to update their test-scoped 
> dependencies for things that moved out of SDK.
> Known examples include protobuf and XML and perhaps others.
> We should re-enable dependency analysis on test scope and manually validate 
> and ignore specific known runtime/test-runtime dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1724) ApexRunner should deduplicate when an UnboundedSource requires Deduping

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1724:
--
Fix Version/s: (was: 2.1.0)

> ApexRunner should deduplicate when an UnboundedSource requires Deduping
> ---
>
> Key: BEAM-1724
> URL: https://issues.apache.org/jira/browse/BEAM-1724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Groh
>
> The Apex implementation of UnboundedRead does not inspect the 
> requiresDeduping property of an Unbounded Source. It should do so.
> https://github.com/apache/beam/blob/master/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2272) Test scope & runtime dependencies need reevaluation

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2272:
--
Fix Version/s: (was: 2.1.0)

> Test scope & runtime dependencies need reevaluation
> ---
>
> Key: BEAM-2272
> URL: https://issues.apache.org/jira/browse/BEAM-2272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
>
> Now that SDK core has been thinned out, many of the modules that have it in 
> their {{dependenciesToScan}} will need to update their test-scoped 
> dependencies for things that moved out of SDK.
> Known examples include protobuf and XML and perhaps others.
> We should re-enable dependency analysis on test scope and manually validate 
> and ignore specific known runtime/test-runtime dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2851

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3835

[Apache Jenkins Server]

See 


Changes:

[tgroh] Translate a Pipeline in SdkComponents

--
[...truncated 721.16 KB...]
2017-05-17T17:56:31.164 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine/1.9.34/appengine-1.9.34.pom
2017-05-17T17:56:31.193 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine/1.9.34/appengine-1.9.34.pom
 (845 B at 27.5 KB/sec)
2017-05-17T17:56:31.195 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/4/oss-parent-4.pom
2017-05-17T17:56:31.222 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/4/oss-parent-4.pom
 (4 KB at 144.7 KB/sec)
2017-05-17T17:56:31.226 [INFO] Downloading: 
https://repository.apache.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-05-17T17:56:31.227 [INFO] Downloading: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-05-17T17:56:31.503 [INFO] Downloaded: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
 (500 B at 1.8 KB/sec)
2017-05-17T17:56:31.505 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.pom
2017-05-17T17:56:31.532 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.pom
 (846 B at 30.6 KB/sec)
2017-05-17T17:56:31.533 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-parent/3.1.2/metrics-parent-3.1.2.pom
2017-05-17T17:56:31.560 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-parent/3.1.2/metrics-parent-3.1.2.pom
 (12 KB at 434.1 KB/sec)
2017-05-17T17:56:31.562 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork26/netty-tcnative-boringssl-static-1.1.33.Fork26.pom
2017-05-17T17:56:31.591 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork26/netty-tcnative-boringssl-static-1.1.33.Fork26.pom
 (17 KB at 581.4 KB/sec)
2017-05-17T17:56:31.593 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-tcnative-parent/1.1.33.Fork26/netty-tcnative-parent-1.1.33.Fork26.pom
2017-05-17T17:56:31.624 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-tcnative-parent/1.1.33.Fork26/netty-tcnative-parent-1.1.33.Fork26.pom
 (20 KB at 636.8 KB/sec)
2017-05-17T17:56:31.626 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.0.18.Final/netty-parent-4.0.18.Final.pom
2017-05-17T17:56:31.655 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.0.18.Final/netty-parent-4.0.18.Final.pom
 (31 KB at 1061.2 KB/sec)
[JENKINS] Archiving disabled
2017-05-17T17:56:35.441 [INFO]  
   
2017-05-17T17:56:35.441 [INFO] 

2017-05-17T17:56:35.441 [INFO] Skipping Apache Beam :: Parent
2017-05-17T17:56:35.441 [INFO] This project has been banned from the build due 
to previous failures.
2017-05-17T17:56:35.441 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled2017-05-17T17:56:51.882 [INFO] 


2017-05-17T17:56:51.882 [INFO] Reactor Summary:

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014485#comment-16014485
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3169


> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3169: [BEAM-1928] Translate a Pipeline in SdkComponents

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3169


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Translate a Pipeline in SdkComponents

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master d6cc850a9 -> 467e68f93


Translate a Pipeline in SdkComponents


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1a05a6a6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1a05a6a6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1a05a6a6

Branch: refs/heads/master
Commit: 1a05a6a66fabfaa2968f81a73df6b8a1a6fc1301
Parents: d6cc850
Author: Thomas Groh 
Authored: Tue Apr 11 16:50:47 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 17 09:29:25 2017 -0700

--
 .../runners/core/construction/PTransforms.java  |  17 ++--
 .../core/construction/SdkComponents.java|  50 ++
 .../core/construction/SdkComponentsTest.java| 100 +++
 3 files changed, 160 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1a05a6a6/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransforms.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransforms.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransforms.java
index 6d2c6b6..d25d342 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransforms.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransforms.java
@@ -66,13 +66,16 @@ public class PTransforms {
   components.registerPCollection((PCollection) 
taggedInput.getValue()));
 }
 for (Map.Entry taggedOutput : 
appliedPTransform.getOutputs().entrySet()) {
-  checkArgument(
-  taggedOutput.getValue() instanceof PCollection,
-  "Unexpected output type %s",
-  taggedOutput.getValue().getClass());
-  transformBuilder.putOutputs(
-  toProto(taggedOutput.getKey()),
-  components.registerPCollection((PCollection) 
taggedOutput.getValue()));
+  // TODO: Remove gating
+  if (taggedOutput.getValue() instanceof PCollection) {
+checkArgument(
+taggedOutput.getValue() instanceof PCollection,
+"Unexpected output type %s",
+taggedOutput.getValue().getClass());
+transformBuilder.putOutputs(
+toProto(taggedOutput.getKey()),
+components.registerPCollection((PCollection) 
taggedOutput.getValue()));
+  }
 }
 for (AppliedPTransform subtransform : subtransforms) {
   
transformBuilder.addSubtransforms(components.getExistingPTransformId(subtransform));

http://git-wip-us.apache.org/repos/asf/beam/blob/1a05a6a6/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
index 2de8237..eb29b9a 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
@@ -22,16 +22,24 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.common.base.Equivalence;
+import com.google.common.collect.ArrayListMultimap;
 import com.google.common.collect.BiMap;
 import com.google.common.collect.HashBiMap;
+import com.google.common.collect.ListMultimap;
 import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Set;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
 import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.TransformHierarchy.Node;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.NameUtils;
 import org.apache.beam.sdk.values.PCollection;
@@ -54,6 +62,48 @@ class SdkComponents {
 return new SdkComponents();
   }
 
+  public static RunnerApi.Pipeline translatePipeline(Pipeline p) {
+final SdkComponents components = create();
+final Collection rootIds = new 

[2/2] beam git commit: This closes #3169

[tgroh]

This closes #3169


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/467e68f9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/467e68f9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/467e68f9

Branch: refs/heads/master
Commit: 467e68f930a626503b5a8b842974fd43ac946b0d
Parents: d6cc850 1a05a6a
Author: Thomas Groh 
Authored: Wed May 17 10:43:12 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 17 10:43:12 2017 -0700

--
 .../runners/core/construction/PTransforms.java  |  17 ++--
 .../core/construction/SdkComponents.java|  50 ++
 .../core/construction/SdkComponentsTest.java| 100 +++
 3 files changed, 160 insertions(+), 7 deletions(-)
--

[jira] [Commented] (BEAM-2185) KafkaIO bounded source

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014439#comment-16014439
 ] 

Raghu Angadi commented on BEAM-2185:


Something like {{withEndTime()}} is required, which sets the upper limit. We 
need a good way to set starting point. https://github.com/apache/beam/pull/3044 
adds {{withStartTime()}. We have all the pieces for someone to work on this.

{{withMaxTime()}} : This might be same as {{withEndTime()}}, If it is walltime, 
it would be problematic with task retries.
{{withMaxRecords()}} : This could be max records per partition, we also need 
{{withStartOffset()}}, also requires roughly uniform partitions.



> KafkaIO bounded source
> --
>
> Key: BEAM-2185
> URL: https://issues.apache.org/jira/browse/BEAM-2185
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>
> KafkaIO could be a useful source for batch applications as well. It could 
> implement a bounded source. The primary question is how the bounds are 
> specified.
> One option : Source specifies a time period (say 9am-10am), and KafkaIO 
> fetches appropriate start and end offsets based on time-index in Kafka. This 
> would suite many batch applications that are launched on a scheduled.
> Another option is to always read till the end and commit the offsets to 
> Kafka. Handling failures and multiple runs of a task might be complicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1872) implement Reshuffle transform

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014360#comment-16014360
 ] 

Ahmet Altay commented on BEAM-1872:
---

Updated link: 
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java

Reshuffle in Java is internal and deprecated, perhaps we can drop this issue.

> implement Reshuffle transform
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>  Labels: newbie, sdk-consistency, starter
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3169: [BEAM-1928] Translate a Pipeline in SdkComponents

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3169

[BEAM-1928] Translate a Pipeline in SdkComponents

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam translate_pipeline_in_components

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3169.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3169


commit 1a05a6a66fabfaa2968f81a73df6b8a1a6fc1301
Author: Thomas Groh 
Date:   2017-04-11T23:50:47Z

Translate a Pipeline in SdkComponents




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #21

[Apache Jenkins Server]

See 


--
[...truncated 1.99 MB...]
2017-05-17T15:51:10.871 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-17T15:51:10.871 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-17T15:51:13.920 [INFO] Replacing original artifact with shaded artifact.
2017-05-17T15:51:13.920 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-17T15:51:13.921 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-17T15:51:13.921 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-17T15:51:14.766 [INFO] 
2017-05-17T15:51:14.766 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-17T15:51:14.947 [INFO] 
2017-05-17T15:51:14.947 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-17T15:51:14.956 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-17T15:51:15.144 [INFO] 
2017-05-17T15:51:15.144 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-17T15:51:15.152 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-17T15:51:15.347 [INFO] 
2017-05-17T15:51:15.347 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-17T15:51:15.863 [INFO] No dependency problems found
2017-05-17T15:51:16.113 [INFO] 
2017-05-17T15:51:16.113 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-17T15:51:16.131 [INFO] Enabled default license matchers.
2017-05-17T15:51:16.131 [INFO] Will parse SCM ignores for exclusions...
2017-05-17T15:51:16.131 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-17T15:51:16.131 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-17T15:51:16.131 [INFO] Exclude: **/target/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-17T15:51:16.131 [INFO] Exclude: **/hs_err_pid*.log
2017-05-17T15:51:16.131 [INFO] Exclude: .github/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/*.iml
2017-05-17T15:51:16.131 [INFO] Exclude: **/.idea/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/*.egg-info/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/package-list
2017-05-17T15:51:16.131 [INFO] Exclude: **/user.avsc
2017-05-17T15:51:16.131 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-17T15:51:16.131 [INFO] Exclude: **/test/**/.placeholder
2017-05-17T15:51:16.131 [INFO] Exclude: .repository/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/nose-*.egg/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/.eggs/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/.tox/**/*
2017-05-17T15:51:16.131 [INFO] Exclude: **/.checkstyle
2017-05-17T15:51:16.131 

[jira] [Created] (BEAM-2311) Update compatibility matrix Aggregators section for Metrics

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-2311:
-

 Summary: Update compatibility matrix Aggregators section for 
Metrics
 Key: BEAM-2311
 URL: https://issues.apache.org/jira/browse/BEAM-2311
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Daniel Halperin
Assignee: Pablo Estrada
 Fix For: Not applicable


CC: [~bchambers] [~aviemzur] [~pabloem]

It looks like the definition of Aggregators needs to be updated post 2.0.0 and 
the switch to Metrics.

https://github.com/apache/beam-site/blob/asf-site/src/_data/capability-matrix.yml#L205



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2310) support data type: DECIMAL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014149#comment-16014149
 ] 

ASF GitHub Bot commented on BEAM-2310:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3168

[BEAM-2310] support new data type: DECIMAL

1. support new data type: DECIMAL
2. support encoding/decoding for TIME
3. add tests all the supported data types.

R: @XuMingmin 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2310-decimal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3168


commit 9a0739dcdfc3b865ea3e74f47bd1188a18abe60a
Author: James Xu 
Date:   2017-05-17T14:48:00Z

support encoding/decoding of TIME

commit 12331d5af6482f18ddfaec2b1e22e192a1605c52
Author: James Xu 
Date:   2017-05-17T14:52:08Z

support new data type: DECIMAL




> support data type: DECIMAL
> --
>
> Key: BEAM-2310
> URL: https://issues.apache.org/jira/browse/BEAM-2310
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3168: [BEAM-2310] support new data type: DECIMAL

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3168

[BEAM-2310] support new data type: DECIMAL

1. support new data type: DECIMAL
2. support encoding/decoding for TIME
3. add tests all the supported data types.

R: @XuMingmin 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2310-decimal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3168


commit 9a0739dcdfc3b865ea3e74f47bd1188a18abe60a
Author: James Xu 
Date:   2017-05-17T14:48:00Z

support encoding/decoding of TIME

commit 12331d5af6482f18ddfaec2b1e22e192a1605c52
Author: James Xu 
Date:   2017-05-17T14:52:08Z

support new data type: DECIMAL




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #247: Add a draft of the Beam FSR blog post

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/247


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: Add a draft of the Beam FSR blog post

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site c4a54639e -> 0b8e34de4


Add a draft of the Beam FSR blog post


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0c8e7595
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0c8e7595
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0c8e7595

Branch: refs/heads/asf-site
Commit: 0c8e75950eef81d73fd103539f6bcdc725478c54
Parents: c4a5463
Author: Dan Halperin 
Authored: Wed May 17 02:31:03 2017 -0400
Committer: Davor Bonaci 
Committed: Wed May 17 04:14:11 2017 -0700

--
 .../2017-05-17-beam-first-stable-release.md | 108 +++
 1 file changed, 108 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/0c8e7595/src/_posts/2017-05-17-beam-first-stable-release.md
--
diff --git a/src/_posts/2017-05-17-beam-first-stable-release.md 
b/src/_posts/2017-05-17-beam-first-stable-release.md
new file mode 100644
index 000..3e97e3b
--- /dev/null
+++ b/src/_posts/2017-05-17-beam-first-stable-release.md
@@ -0,0 +1,108 @@
+---
+layout: post
+title:  "Apache Beam publishes the first stable release"
+date:   2017-05-17 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+  - davor
+  - dhalperi
+---
+
+The Apache Beam community is pleased to [announce the availability of version 
2.0.0](https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces12).
 This is the first stable release of Apache Beam, signifying a statement from 
the community that it intends to maintain API stability with all releases for 
the foreseeable future, and making Beam suitable for enterprise deployment.
+
+
+
+This first stable release is the third important milestone for the Apache Beam 
community. Beam joined the Apache Incubator in February 2016 and graduated as a 
top-level project of The Apache Software Foundation in December. Through these 
fifteen months of concentrated effort, a slightly chaotic codebase, merged from 
multiple organizations, has been developed into a generalized framework for 
data processing that is truly engine- and environment-independent. Apache Beam 
has evolved and improved through three incubating and three post-incubation 
releases, culminating in the stable release announced today as version 2.0.0.
+ 
+In the five months since graduation, Apache Beam has seen a significant 
growth, both in terms of adoption and community contribution. Apache Beam is 
[in 
use](https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces12)
 at Google Cloud, PayPal, and Talend, among others.
+ 
+Apache Beam, version 2.0.0 improves user experience across the project, 
focusing on seamless portability across execution environments, including 
engines, operating systems, on-premise clusters, cloud providers, and data 
storage systems. Other highlights include:
+
+* API stability and future compatibility within this major version.
+* Stateful data processing paradigms that unlock efficient, data-dependent 
computations.
+* Support for user-extensible file systems, with built-in support for Hadoop 
Distributed File System, among others.
+* A metrics subsystem for deeper insight into pipeline execution.
+
+Many contributors made this release possible, by participating in different 
roles: contributing code, writing documentation, testing release candidates, 
supporting users, or helping in some other way. The following is a partial list 
of contributors – 76 individuals contributed code to the project since the 
previous release, assembled from source history:
+
+* Ahmet Altay
+* Eric Anderson
+* Raghu Angadi
+* Sourabh Bajaj
+* Péter Gergő Barna
+* Chen Bin
+* Davor Bonaci
+* Robert Bradshaw
+* Ben Chambers
+* Etienne Chauchot
+* Chang Chen
+* Charles Chen
+* Craig Citro
+* Lukasz Cwik
+* Márton Elek
+* Pablo Estrada
+* Josh Forman-Gornall
+* Maria García Herrero
+* Jins George
+* Damien Gouyette
+* Thomas Groh
+* Dan Halperin
+* Pei He
+* Hadar Hod
+* Chamikara Jayalath
+* Rekha Joshi
+* Uwe Jugel
+* Sung Junyoung
+* Holden Karau
+* Vikas Kedigehalli
+* Eugene Kirpichov
+* Tibor Kiss
+* Kenneth Knowles
+* Vassil Kolarov
+* Chinmay Kolhatkar
+* Aljoscha Krettek
+* Dipti Kulkarni
+* Radhika Kulkarni
+* Jason Kuster
+* Reuven Lax
+* Stas Levin
+* Julien Lhermitte
+* Jingsong Li
+* Neville Li
+* Mark Liu
+* Michael Luckey
+* Andrew Martin
+* Ismaël Mejía
+* Devon Meunier
+* Neda Mirian
+* Anil Muppalla
+* Gergely Novak
+* Jean-Baptiste Onofré
+* Melissa Pashniak
+* peay
+* David Rieber
+* Rahul Sabbineni
+* Kobi Salant
+* Amit Sela
+* Mark Shalda
+* Stephen Sisk
+* Yuya Tajima
+* Wesley Tanaka
+* JiJun Tang
+* Valentyn Tymofieiev
+* David 

[3/3] beam-site git commit: This closes #247

[davor]

This closes #247


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0b8e34de
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0b8e34de
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0b8e34de

Branch: refs/heads/asf-site
Commit: 0b8e34de459ebf4a915c5ae68f5a683b80a05501
Parents: c4a5463 0b67829
Author: Davor Bonaci 
Authored: Wed May 17 04:14:58 2017 -0700
Committer: Davor Bonaci 
Committed: Wed May 17 04:14:58 2017 -0700

--
 .../2017/05/17/beam-first-stable-release.html   | 308 +++
 content/blog/index.html |  16 +
 content/feed.xml| 148 ++---
 content/index.html  |  10 +-
 .../2017-05-17-beam-first-stable-release.md | 108 +++
 5 files changed, 550 insertions(+), 40 deletions(-)
--

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0b678294
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0b678294
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0b678294

Branch: refs/heads/asf-site
Commit: 0b678294a57237842a57e7afa86c31f5c8eb1e55
Parents: 0c8e759
Author: Davor Bonaci 
Authored: Wed May 17 04:14:58 2017 -0700
Committer: Davor Bonaci 
Committed: Wed May 17 04:14:58 2017 -0700

--
 .../2017/05/17/beam-first-stable-release.html   | 308 +++
 content/blog/index.html |  16 +
 content/feed.xml| 148 ++---
 content/index.html  |  10 +-
 4 files changed, 442 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/0b678294/content/blog/2017/05/17/beam-first-stable-release.html
--
diff --git a/content/blog/2017/05/17/beam-first-stable-release.html 
b/content/blog/2017/05/17/beam-first-stable-release.html
new file mode 100644
index 000..fabace3
--- /dev/null
+++ b/content/blog/2017/05/17/beam-first-stable-release.html
@@ -0,0 +1,308 @@
+
+
+  
+  
+  
+  
+  Apache Beam publishes the first stable release
+  
+  https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  https://beam.apache.org/blog/2017/05/17/beam-first-stable-release.html; 
data-proofer-ignore>
+  
+  https://beam.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+  
+
+
+  
+
+
+  
+
+  
+
+  Toggle navigation
+  
+  
+  
+
+
+
+  
+
+  Get Started 
+  
+Beam Overview
+Quickstart - 
Java
+Quickstart - 
Python
+
+Example Walkthroughs
+WordCount
+Mobile 
Gaming
+
+Resources
+Downloads
+Support
+  
+
+
+  Documentation 
+  
+Using the Documentation
+
+Beam Concepts
+Programming 
Guide
+Additional 
Resources
+
+Pipeline Fundamentals
+Design Your 
Pipeline
+Create Your 
Pipeline
+Test 
Your Pipeline
+Pipeline I/O
+
+SDKs
+Java SDK
+Java SDK API Reference 
+
+Python SDK
+Python SDK API Reference 
+
+
+Runners
+Capability 
Matrix
+Direct Runner
+Apache Apex 
Runner
+Apache Flink 
Runner
+Apache Spark 
Runner
+Cloud Dataflow 
Runner
+  
+
+
+  Contribute 
+  
+Get Started Contributing
+
+Guides
+Contribution 
Guide
+Testing Guide
+Release Guide
+PTransform Style 
Guide
+
+Technical References
+Design 
Principles
+Ongoing 
Projects
+Source 
Repository
+
+Promotion
+Presentation 
Materials
+Logos and Design
+
+Maturity Model
+Team
+  
+
+
+Blog
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:20px;">
+  
+http://www.apache.org/;>ASF Homepage
+http://www.apache.org/licenses/;>License
+http://www.apache.org/security/;>Security
+http://www.apache.org/foundation/thanks.html;>Thanks
+http://www.apache.org/foundation/sponsorship.html;>Sponsorship
+https://www.apache.org/foundation/policies/conduct;>Code of 
Conduct
+  
+
+  
+
+
+
+
+  
+
+http://schema.org/BlogPosting;>
+
+  
+Apache Beam publishes the 
first stable release
+May 17, 2017 •  Davor Bonaci [https://twitter.com/BonaciDavor;>@BonaciDavor]  Dan Halperin 
+
+  
+
+  
+The Apache Beam community is pleased to 

[jira] [Commented] (BEAM-1507) java DataflowRunner should warn if the stagingLocation has a TTL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013831#comment-16013831
 ] 

ASF GitHub Bot commented on BEAM-1507:
--

GitHub user hepeknet opened a pull request:

https://github.com/apache/beam/pull/3167

[BEAM-1507] adding TTL check for staging location

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-1507] adding TTL check for staging location`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

@dhalperi you like this solution?
---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hepeknet/beam BEAM-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3167.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3167


commit 01f2c894de0f80e5f7830f1301e8d5133e5bfd7e
Author: Borisa Zivkovic 
Date:   2017-05-17T10:37:03Z

adding TTL check




> java DataflowRunner should warn if the stagingLocation has a TTL
> 
>
> Key: BEAM-1507
> URL: https://issues.apache.org/jira/browse/BEAM-1507
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Priority: Minor
>  Labels: newbie, starter
>
> We have seen a few customers run into a hard-to-track-down bug where the 
> staging bucket has a TTL, but files get TTL-deleted when they are still 
> needed.
> This might be because of:
> 1. Long lived batch jobs / streaming jobs can reference staged files 
> arbitrarily later and will fail in bad ways if they have been deleted.
> 2. Some customers even hit issues where the "check file already exists" 
> succeeds when starting a job, but then the file is TTL-deleted before the job 
> actually starts. (This sounds crazy, but may happen if TTL is 7 days and jobs 
> run every 7 days, for example. Race condition.)
> I'm hoping it's not hard to check that files would have TTLs and warn if so.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3167: [BEAM-1507] adding TTL check for staging location

[hepeknet]

GitHub user hepeknet opened a pull request:

https://github.com/apache/beam/pull/3167

[BEAM-1507] adding TTL check for staging location

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-1507] adding TTL check for staging location`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

@dhalperi you like this solution?
---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hepeknet/beam BEAM-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3167.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3167


commit 01f2c894de0f80e5f7830f1301e8d5133e5bfd7e
Author: Borisa Zivkovic 
Date:   2017-05-17T10:37:03Z

adding TTL check




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2310) support data type: DECIMAL

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2310:
--

Assignee: (was: Xu Mingmin)

> support data type: DECIMAL
> --
>
> Key: BEAM-2310
> URL: https://issues.apache.org/jira/browse/BEAM-2310
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2310) support data type: DECIMAL

[James Xu (JIRA)]

James Xu created BEAM-2310:
--

 Summary: support data type: DECIMAL
 Key: BEAM-2310
 URL: https://issues.apache.org/jira/browse/BEAM-2310
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: James Xu
Assignee: Xu Mingmin






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3166: [BEAM-2309] implement VALUES

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3166

[BEAM-2309] implement VALUES

1. implement VALUES
2. add support for data type CHAR(to be able to test VALUES)

R: @XuMingmin 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2309-values

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3166.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3166


commit 72217a0ff8d75842e33bf98b5cf91b258725f1c8
Author: James Xu 
Date:   2017-05-17T06:41:18Z

implement VALUES

1. implement VALUES
2. add support for data type CHAR(to be able to test VALUES)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

beam-site git commit: Regenerate website after merge

[dhalperi]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 222ba1259 -> c4a54639e


Regenerate website after merge


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/c4a54639
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/c4a54639
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/c4a54639

Branch: refs/heads/asf-site
Commit: c4a54639e0a25ba74c5fd5eb62eeab371f6e614d
Parents: 222ba12
Author: Dan Halperin 
Authored: Wed May 17 02:33:04 2017 -0400
Committer: Dan Halperin 
Committed: Wed May 17 02:33:04 2017 -0400

--
 content/feed.xml   | 2 +-
 content/get-started/quickstart-java/index.html | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/c4a54639/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 02da655..1a32bd5 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -883,7 +883,7 @@ be controlled within a test./p
 h2 
id=writing-deterministic-tests-to-emulate-nondeterminismWriting 
Deterministic Tests to Emulate Nondeterminism/h2
 
 pThe Beam testing infrastructure provides the
-a 
href=/documentation/sdks/javadoc/0.6.0/org/apache/beam/sdk/testing/PAssert.htmlPAssert/a
+a 
href=/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/testing/PAssert.htmlPAssert/a
 methods, which assert properties about the contents of a PCollection from 
within
 a pipeline. We have expanded this infrastructure to include
 a 
href=https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.javaTestStream/a;,

http://git-wip-us.apache.org/repos/asf/beam-site/blob/c4a54639/content/get-started/quickstart-java/index.html
--
diff --git a/content/get-started/quickstart-java/index.html 
b/content/get-started/quickstart-java/index.html
index abee09e..0d9d0af 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -164,10 +164,9 @@
 The easiest way to get a copy of the WordCount pipeline is to use the 
following command to generate a simple Maven project that contains Beam’s 
WordCount examples and builds against the most recent Beam release:
 
 $ mvn 
archetype:generate \
-  
-DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
   -DarchetypeGroupId=org.apache.beam \
   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-  -DarchetypeVersion=LATEST \
+  -DarchetypeVersion=2.0.0 \
   -DgroupId=org.example \
   -DartifactId=word-count-beam \
   -Dversion="0.1" \

[GitHub] beam-site pull request #247: Add a draft of the Beam FSR blog post

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam-site/pull/247

Add a draft of the Beam FSR blog post

R: @davorbonaci 

I played with tables for 10 minutes but I realized that the existing tables 
are YAML+HTML (cap matrix) and decided this was enough for now.

Missing links.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam-site fsr-blog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #247


commit c3aff52c11aaa6099327beedfcfe9b44c3f3ffae
Author: Dan Halperin 
Date:   2017-05-17T06:31:03Z

Add a draft of the Beam FSR blog post




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---