[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005912#comment-16005912
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3074

[BEAM-1340] Adds __all__ tags to classes in package apache_beam/io

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_public_api_all

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3074


commit b5ff3ba87869aab31eb502d039c853c46e7ff818
Author: chamik...@google.com 
Date:   2017-05-11T05:33:35Z

Adds __all__ tags to classes in package apache_beam/io.




> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3074: [BEAM-1340] Adds __all__ tags to classes in package...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3074

[BEAM-1340] Adds __all__ tags to classes in package apache_beam/io

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_public_api_all

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3074


commit b5ff3ba87869aab31eb502d039c853c46e7ff818
Author: chamik...@google.com 
Date:   2017-05-11T05:33:35Z

Adds __all__ tags to classes in package apache_beam/io.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2222) Clean up readme files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005910#comment-16005910
 ] 

ASF GitHub Bot commented on BEAM-:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/236


> Clean up readme files
> -
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: 2.0.0
>
>
> Move content from readme.md's to website.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/d5140a40
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/d5140a40
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/d5140a40

Branch: refs/heads/asf-site
Commit: d5140a4096a9958d9d674bf80eb9f24f0a59006b
Parents: 212b9c7
Author: Ahmet Altay 
Authored: Wed May 10 22:33:17 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 10 22:33:17 2017 -0700

--
 content/documentation/io/built-in/hadoop/index.html | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/d5140a40/content/documentation/io/built-in/hadoop/index.html
--
diff --git a/content/documentation/io/built-in/hadoop/index.html 
b/content/documentation/io/built-in/hadoop/index.html
index 7fda51a..c5f051a 100644
--- a/content/documentation/io/built-in/hadoop/index.html
+++ b/content/documentation/io/built-in/hadoop/index.html
@@ -219,7 +219,7 @@
 
 Read data with 
configuration and key translation
 
-For example scenario: Beam Coder is 
not available for key class hence key translation is required.
+For example, a Beam Coder is not 
available for Key class, so key 
translation is required.
 
 p.apply("read",
   HadoopInputFormatIO.MyKeyClass, InputFormatKeyClassread()
@@ -234,7 +234,7 @@
 
 Read data with 
configuration and value translation
 
-For example scenario: Beam Coder is 
not available for value class hence value translation is required.
+For example, a Beam Coder is not 
available for Value class, so value 
translation is required.
 
 p.apply("read",
   HadoopInputFormatIO.InputFormatKeyClass, MyValueClassread()
@@ -249,7 +249,7 @@
 
 Read 
data with configuration, value translation and key translation
 
-For example scenario: Beam Coders are not available for both Key class and Value class of InputFormat hence key and value translation is 
required.
+For example, Beam Coders are not available for both Key class and Value classes of InputFormat, so key and value translation are 
required.
 
 p.apply("read",
   HadoopInputFormatIO.MyKeyClass, MyValueClassread()

[GitHub] beam-site pull request #236: [BEAM-2222] fix typos in hadoop documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/236


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: fix typos in hadoop io documentation

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 32241b501 -> d42cbaf11


fix typos in hadoop io documentation


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/212b9c7d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/212b9c7d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/212b9c7d

Branch: refs/heads/asf-site
Commit: 212b9c7d56748e5a7c025784394cf3c4e61481d9
Parents: 32241b5
Author: Ahmet Altay 
Authored: Wed May 10 10:49:25 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 10 22:29:46 2017 -0700

--
 src/documentation/io/built-in-hadoop.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/212b9c7d/src/documentation/io/built-in-hadoop.md
--
diff --git a/src/documentation/io/built-in-hadoop.md 
b/src/documentation/io/built-in-hadoop.md
index dda5843..5c07717 100644
--- a/src/documentation/io/built-in-hadoop.md
+++ b/src/documentation/io/built-in-hadoop.md
@@ -70,7 +70,7 @@ p.apply("read",
 
  Read data with configuration and key translation 
 
-For example scenario: Beam `Coder` is not available for key class hence key 
translation is required.
+For example, a Beam `Coder` is not available for `Key` class, so key 
translation is required.
 
 ```java
 p.apply("read",
@@ -85,7 +85,7 @@ p.apply("read",
 
  Read data with configuration and value translation
 
-For example scenario: Beam `Coder` is not available for value class hence 
value translation is required.
+For example, a Beam `Coder` is not available for `Value` class, so value 
translation is required.
 
 ```java
 p.apply("read",
@@ -100,7 +100,7 @@ p.apply("read",
 
  Read data with configuration, value translation and key translation 
 
-For example scenario: Beam Coders are not available for both `Key` class and 
`Value` class of `InputFormat` hence key and value translation is required.
+For example, Beam Coders are not available for both `Key` class and `Value` 
classes of `InputFormat`, so key and value translation are required.
 
 ```java
 p.apply("read",

[3/3] beam-site git commit: This closes #236

[altay]

This closes #236


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/d42cbaf1
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/d42cbaf1
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/d42cbaf1

Branch: refs/heads/asf-site
Commit: d42cbaf119c324926daf06fc3ecdf38bd7fc5fd1
Parents: 32241b5 d5140a4
Author: Ahmet Altay 
Authored: Wed May 10 22:33:17 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 10 22:33:17 2017 -0700

--
 content/documentation/io/built-in/hadoop/index.html | 6 +++---
 src/documentation/io/built-in-hadoop.md | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)
--

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3771

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2194

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1439

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #2036

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3073: [BEAM-2246] Use CLIENT_ACK instead of AUTO_ACK in J...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam/pull/3073

[BEAM-2246] Use CLIENT_ACK instead of AUTO_ACK in JmsIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam BEAM-2246-JMS-SESSION

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3073.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3073


commit de00c58808902d5d4593fba41f637884df67c90b
Author: Jean-Baptiste Onofré 
Date:   2017-05-10T05:39:56Z

[BEAM-2246] Use CLIENT_ACK instead of AUTO_ACK in JmsIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2246) JmsIO should use CLIENT_ACK instead of AUTO_ACK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005880#comment-16005880
 ] 

ASF GitHub Bot commented on BEAM-2246:
--

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam/pull/3073

[BEAM-2246] Use CLIENT_ACK instead of AUTO_ACK in JmsIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam BEAM-2246-JMS-SESSION

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3073.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3073


commit de00c58808902d5d4593fba41f637884df67c90b
Author: Jean-Baptiste Onofré 
Date:   2017-05-10T05:39:56Z

[BEAM-2246] Use CLIENT_ACK instead of AUTO_ACK in JmsIO




> JmsIO should use CLIENT_ACK instead of AUTO_ACK
> ---
>
> Key: BEAM-2246
> URL: https://issues.apache.org/jira/browse/BEAM-2246
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> As {{JmsIO}} manages the messages acknowledgement in its checkpoint, it 
> should not use {{AUTO_ACKNOWLEDGE}} but {{CLIENT_ACKNOWLEDGE}}.
> With {{AUTO_ACK}}, the ack is sent to the JMS broker as soon as the message 
> is consumed. Basically, it means that the checkpoint is useless as, if 
> something goes wrong in the pipeline, the messages is already consumed from 
> the broker (so no way to redeliver the message).
> Instead, {{JmsIO}} should use {{CLIENT_ACK}} and explicitly send the ack 
> (commit) in the checkpoint (the code is already in place). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #231: Website updates for version 2.0.0: downloads, J...

[davorbonaci]

Github user davorbonaci closed the pull request at:

https://github.com/apache/beam-site/pull/231


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2783

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2035

[Apache Jenkins Server]

See 


--
Started by user dhalperi
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2782

[Apache Jenkins Server]

See 


--
Started by user dhalperi
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

Build failed in Jenkins: beam_PostCommit_Python_Verify #2193

[Apache Jenkins Server]

See 


--
Started by user dhalperi
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3770

[Apache Jenkins Server]

See 


--
Started by user dhalperi
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1438

[Apache Jenkins Server]

See 


--
Started by user dhalperi
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/2.0.0-RC2^{commit} # timeout=10
 > git rev-parse 2.0.0-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

[GitHub] beam-site pull request #240: Release guide: fix instructions

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam-site/pull/240

Release guide: fix instructions

R: @aaltay 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam-site fix-release-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #240


commit ad7f02c5a9f7ee7a3b9d02acc35101cd117f2fd9
Author: Dan Halperin 
Date:   2017-05-11T03:00:16Z

Release guide: fix instructions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #239: Website updates for version 2.0.0 (rc2): downlo...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam-site/pull/239

Website updates for version 2.0.0 (rc2): downloads, Javadoc, Pydoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam-site 2.0.0-rc2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #239


commit a7c78c60110c35a0b2860d35121f1e6efed39e81
Author: Davor Bonaci 
Date:   2017-05-07T03:23:53Z

Update latest version metadata information

commit 48322b0e22cb1c550268ba1aeec9da3af69faab6
Author: Davor Bonaci 
Date:   2017-05-07T03:28:31Z

Add version 2.0.0 to Downloads page

commit 759104c52fce2152c8add65cb22ff6c4027e23af
Author: Dan Halperin 
Date:   2017-05-11T02:52:33Z

Add Javadoc for 2.0.0 (rc#2)

commit dacb4364d41ff253d91ba5cb88e9e58a3fbf5c89
Author: Dan Halperin 
Date:   2017-05-11T02:54:10Z

Add Pydoc for 2.0.0 (rc #2)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-2150) Support for recursive wildcards in GcsPath

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-2150.
-
   Resolution: Fixed
Fix Version/s: 2.1.0

> Support for recursive wildcards in GcsPath
> --
>
> Key: BEAM-2150
> URL: https://issues.apache.org/jira/browse/BEAM-2150
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Devon Meunier
>Assignee: Devon Meunier
>Priority: Minor
> Fix For: 2.1.0
>
>
> When working with heavily nested folder structures in Google Cloud Storage, 
> it's great to make use of recursive wildcards, which the current API 
> explicitly does not support.
> This code hasn't been touched in 2 years so it's likely that simply no one's 
> gotten around to it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2150) Support for recursive wildcards in GcsPath

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005788#comment-16005788
 ] 

ASF GitHub Bot commented on BEAM-2150:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2866


> Support for recursive wildcards in GcsPath
> --
>
> Key: BEAM-2150
> URL: https://issues.apache.org/jira/browse/BEAM-2150
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Devon Meunier
>Assignee: Devon Meunier
>Priority: Minor
>
> When working with heavily nested folder structures in Google Cloud Storage, 
> it's great to make use of recursive wildcards, which the current API 
> explicitly does not support.
> This code hasn't been touched in 2 years so it's likely that simply no one's 
> gotten around to it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-2150] Relax regex to support wildcard globbing for GCS

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 62ee27542 -> 15bd3a394


[BEAM-2150] Relax regex to support wildcard globbing for GCS


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/316ff6bf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/316ff6bf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/316ff6bf

Branch: refs/heads/master
Commit: 316ff6bf9f472687667482805062b0aa6417aa8f
Parents: 62ee275
Author: Devon Meunier 
Authored: Wed May 3 16:22:16 2017 -0400
Committer: Dan Halperin 
Committed: Wed May 10 18:59:30 2017 -0700

--
 .../extensions/gcp/storage/GcsFileSystem.java   |  6 +--
 .../gcp/storage/GcsPathValidator.java   |  4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  | 46 +---
 .../gcp/storage/GcsFileSystemTest.java  |  8 ---
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 57 
 5 files changed, 63 insertions(+), 58 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/316ff6bf/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
--
diff --git 
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
 
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
index 9052a5a..6db0a01 100644
--- 
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
+++ 
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
@@ -73,7 +73,7 @@ class GcsFileSystem extends FileSystem {
 List isGlobBooleans = Lists.newArrayList();
 
 for (GcsPath path : gcsPaths) {
-  if (GcsUtil.isGlob(path)) {
+  if (GcsUtil.isWildcard(path)) {
 globs.add(path);
 isGlobBooleans.add(true);
   } else {
@@ -178,8 +178,8 @@ class GcsFileSystem extends FileSystem {
*/
   @VisibleForTesting
   MatchResult expand(GcsPath gcsPattern) throws IOException {
-String prefix = GcsUtil.getGlobPrefix(gcsPattern.getObject());
-Pattern p = Pattern.compile(GcsUtil.globToRegexp(gcsPattern.getObject()));
+String prefix = GcsUtil.getNonWildcardPrefix(gcsPattern.getObject());
+Pattern p = 
Pattern.compile(GcsUtil.wildcardToRegexp(gcsPattern.getObject()));
 
 LOG.debug("matching files in bucket {}, prefix {} against pattern {}", 
gcsPattern.getBucket(),
 prefix, p.toString());

http://git-wip-us.apache.org/repos/asf/beam/blob/316ff6bf/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
--
diff --git 
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
 
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
index e7257b2..6b10bfb 100644
--- 
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
+++ 
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
@@ -23,7 +23,6 @@ import java.io.IOException;
 import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
 import org.apache.beam.sdk.io.fs.ResourceId;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.GcsUtil;
 import org.apache.beam.sdk.util.gcsfs.GcsPath;
 
 /**
@@ -47,8 +46,7 @@ public class GcsPathValidator implements PathValidator {
*/
   @Override
   public void validateInputFilePatternSupported(String filepattern) {
-GcsPath gcsPath = getGcsPath(filepattern);
-checkArgument(GcsUtil.isGcsPatternSupported(gcsPath.getObject()));
+getGcsPath(filepattern);
 verifyPath(filepattern);
 verifyPathIsAccessible(filepattern, "Could not find file %s");
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/316ff6bf/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
--
diff --git 
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
 
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
index 18e3e2b..94b733a 100644
--- 

[GitHub] beam pull request #2866: [BEAM-2150] Relax regex to support wildcard globbin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2866


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2866

[dhalperi]

This closes #2866


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/15bd3a39
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/15bd3a39
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/15bd3a39

Branch: refs/heads/master
Commit: 15bd3a394e5c724216453a2098af117cac13778e
Parents: 62ee275 316ff6b
Author: Dan Halperin 
Authored: Wed May 10 18:59:33 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 18:59:33 2017 -0700

--
 .../extensions/gcp/storage/GcsFileSystem.java   |  6 +--
 .../gcp/storage/GcsPathValidator.java   |  4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  | 46 +---
 .../gcp/storage/GcsFileSystemTest.java  |  8 ---
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 57 
 5 files changed, 63 insertions(+), 58 deletions(-)
--

svn commit: r19615 - /dev/beam/2.0.0-RC2/

[dhalperi]

Author: dhalperi
Date: Thu May 11 01:38:50 2017
New Revision: 19615

Log:
Apache Beam 2.0.0, release candidate 2

Added:
dev/beam/2.0.0-RC2/
dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip   (with props)
dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.asc
dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.md5
dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.sha1
dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip   (with props)
dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.asc
dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.md5
dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.sha1

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.asc
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.asc (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.asc Thu May 11 01:38:50 2017
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQEcBAABAgAGBQJZE8CIAAoJEGrhtLGEXmaJOwYIALOE0L4HdPG5UsyG5fPXgBDi
+ty20r0ohwyywIcOTLXA28MDXSiRk+hiiCteGlKoUhdovjaglyPvOIxu2paKkxtxu
+O5+GSrqEwYNvcAJ2JhdtR90V/QCavaVtqF+8DqJ7kFFtkvLtKiIch/g+mQ8FBC3t
+lghCShEEy61r1iiH5xoMRu+OtX5hvnA/SKe1BTHKFkHIXq4R6OjUbqsJOMn4f10x
+D40DVeV6s9v0Dl4Z3Xtu0VbwFjqUguZtlHBEfZjoFIo2N/iekR3DJaa5UOeYartQ
+dX8JKkgAVAp/ges5ICsIRN0Bpqx1zi2HLprAPBAF2Tm95L5tN2J9k0VJBdKscUQ=
+=JzUz
+-END PGP SIGNATURE-

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.md5
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.md5 (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.md5 Thu May 11 01:38:50 2017
@@ -0,0 +1 @@
+0ce727071942bfdfa7c4543cbeefcb6d  apache-beam-2.0.0-python.zip

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.sha1
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.sha1 (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-python.zip.sha1 Thu May 11 01:38:50 
2017
@@ -0,0 +1 @@
+25f5da51130d1e901a7ad4e7f2a3ba62b746285c  apache-beam-2.0.0-python.zip

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.asc
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.asc (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.asc Thu May 11 
01:38:50 2017
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQEcBAABAgAGBQJZE75+AAoJEGrhtLGEXmaJuSQH/0+6VQ7ZdpA/EWsfiFu4vWrA
+ddEMhRD8Ju9TrEghYO2aQNgWmPsc4lBzX9P/S+94tR77IdayvykI+DcZ5iHP8Vyt
+oASbjXfd2w2CRkJzZMjkfp6NQ/mdJEDuWr4Dah0bLlD1DW1p+FSld9H6ua0T7HTf
+QasMx8f8W+l5JGzh0Tww3etD+MDHvI+iMNrY9FTAR33KaOIP5iYZo9AQ0J8Dsysf
+GsX1h1r6RjwQjPHZcrCj9cRm4nEW49DjGL8qJFKWeYc81dSCR8pdt1RBCdsUXMGd
+sMJXUNLH7r8oz/Ebi34F2q9NBZjHTjrRth9QJyd/en6dXXG3AATSpHneFDZF0JA=
+=XkIn
+-END PGP SIGNATURE-

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.md5
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.md5 (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.md5 Thu May 11 
01:38:50 2017
@@ -0,0 +1 @@
+bb8c0b3d4af3e4d528e4cf6b45921623  apache-beam-2.0.0-source-release.zip

Added: dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.sha1
==
--- dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.sha1 (added)
+++ dev/beam/2.0.0-RC2/apache-beam-2.0.0-source-release.zip.sha1 Thu May 11 
01:38:50 2017
@@ -0,0 +1 @@
+d152f7829766d67647766f8268ed270b08035fea  apache-beam-2.0.0-source-release.zip

beam git commit: [maven-release-plugin] prepare release v2.0.0-RC2

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 7485583a3 -> 9778c21f9


[maven-release-plugin] prepare release v2.0.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9778c21f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9778c21f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9778c21f

Branch: refs/heads/release-2.0.0
Commit: 9778c21f9b0720073799c78e3be11d1c97bed766
Parents: 7485583
Author: Dan Halperin 
Authored: Wed May 10 18:36:40 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 18:36:40 2017 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 4 ++--
 runners/flink/pom.xml   | 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/common/fn-api/pom.xml  | 2 +-
 sdks/common/pom.xml | 2 +-
 sdks/common/runner-api/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml| 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml| 2 +-
 sdks/java/io/hadoop/pom.xml | 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml| 2 +-
 sdks/java/io/mongodb/pom.xml| 2 +-
 sdks/java/io/mqtt/pom.xml   | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/io/xml/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/javadoc/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 sdks/python/pom.xml | 2 +-
 50 files changed, 52 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9778c21f/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index cb566d6..495f86a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.0.0-SNAPSHOT
+2.0.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/9778c21f/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index db79292..86f870b 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.0.0-SNAPSHOT
+2.0.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/9778c21f/examples/pom.xml

[beam] Git Push Summary

[dhalperi]

Repository: beam
Updated Tags:  refs/tags/v2.0.0-RC2 [created] 97c4a09ab

beam git commit: [maven-release-plugin] rollback changes from release preparation of v2.0.0-RC2

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 9778c21f9 -> 66b3ce075


[maven-release-plugin] rollback changes from release preparation of v2.0.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/66b3ce07
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/66b3ce07
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/66b3ce07

Branch: refs/heads/release-2.0.0
Commit: 66b3ce0754f07974981a584467ebb271fc8e32be
Parents: 9778c21
Author: Dan Halperin 
Authored: Wed May 10 18:36:54 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 18:36:54 2017 -0700

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/common/fn-api/pom.xml  | 2 +-
 sdks/common/pom.xml | 2 +-
 sdks/common/runner-api/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml| 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml| 2 +-
 sdks/java/io/hadoop/pom.xml | 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml| 2 +-
 sdks/java/io/mongodb/pom.xml| 2 +-
 sdks/java/io/mqtt/pom.xml   | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/io/xml/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/javadoc/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 sdks/python/pom.xml | 2 +-
 50 files changed, 51 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/66b3ce07/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 495f86a..cb566d6 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.0.0
+2.0.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/66b3ce07/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 86f870b..db79292 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.0.0
+2.0.0-SNAPSHOT
 ../pom.xml
   
 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2780

[Apache Jenkins Server]

See 

[1/2] beam git commit: bump time of precommits

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master a39960b13 -> 62ee27542


bump time of precommits

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/416b89b7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/416b89b7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/416b89b7

Branch: refs/heads/master
Commit: 416b89b7ff1bd26e7d9a1b3c29f294d2ae922238
Parents: a39960b
Author: Jason Kuster 
Authored: Wed May 10 17:55:53 2017 -0700
Committer: Jason Kuster 
Committed: Wed May 10 17:55:53 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/416b89b7/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
index 3718551..bc130ec 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
@@ -29,7 +29,10 @@ mavenJob('beam_PreCommit_Java_MavenInstall') {
   concurrentBuild()
 
   // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(delegate)
+  common_job_properties.setTopLevelMainJobProperties(
+delegate,
+'master',
+120)
 
   // Set Maven parameters.
   common_job_properties.setMavenConfig(delegate)

[GitHub] beam pull request #3071: bump time of precommits

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3071


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3071: bump time of precommits

[kenn]

This closes #3071: bump time of precommits


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/62ee2754
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/62ee2754
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/62ee2754

Branch: refs/heads/master
Commit: 62ee27542866189f8726cc7b047a1ed55503cb44
Parents: a39960b 416b89b
Author: Kenneth Knowles 
Authored: Wed May 10 18:03:13 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 18:03:13 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--

[GitHub] beam pull request #3072: Remove protobuf and http-client dependency from Dat...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3072

Remove protobuf and http-client dependency from Dataflow runner

runners/google-cloud-dataflow

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam pbf

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3072.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3072


commit 8be5466567dfe79fb7382c56cee44ab5c39ac01c
Author: Vikas Kedigehalli 
Date:   2017-05-11T00:39:01Z

Remove protobuf and http-client dependency from
runners/google-cloud-dataflow




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3028: Shade protobuf-java in runners/google-cloud-dataflo...

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/beam/pull/3028


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3071: bump time of precommits

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/3071

bump time of precommits

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam bump-time

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3071.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3071


commit 416b89b7ff1bd26e7d9a1b3c29f294d2ae922238
Author: Jason Kuster 
Date:   2017-05-11T00:55:53Z

bump time of precommits

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3070: [BEAM-3065] Mark utils/* as internal

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3070

[BEAM-3065] Mark utils/* as internal

R: @robertwb 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam clean

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3070.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3070


commit 45a5733ed6354308226e1fe0d2a4194929a50e16
Author: Ahmet Altay 
Date:   2017-05-11T00:38:53Z

Mark utils/* as internal




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2779

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-2185) KafkaIO bounded source

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005711#comment-16005711
 ] 

Raghu Angadi edited comment on BEAM-2185 at 5/11/17 12:23 AM:
--

Some more considerations : 
  - Splits:
 -- Each partition should be split further. 
 -- If we could fetch the size in bytes, it is easier to split. I don't 
think it is feasible get byte offsets.
 -- Split could be based on message offsets, but the size could be too 
large or too small depending on average message size.
 -- Take a hint from the user if the size is not known? May be not. Read a 
few sample records? Probably an over kill. 
  - It should support liquid-sharding (dynamic splitting). This does not need 
average message size. Just the offsets are good enough.
  


was (Author: rangadi):
Some more considerations : 
  - Splits:
 - Each partition should be split further. 
 - If we could fetch the size in bytes, it is easier to split. I don't 
think it is feasible get byte offsets.
 - Split could be based on message offsets, but the size could be too large 
or too small depending on average message size.
 - Take a hint from the user if the size is not known? May be not. Read a 
few sample records? Probably an over kill. 
  - It should support liquid-sharding (dynamic splitting). This does not need 
average message size. Just the offsets are good enough.
  

> KafkaIO bounded source
> --
>
> Key: BEAM-2185
> URL: https://issues.apache.org/jira/browse/BEAM-2185
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>
> KafkaIO could be a useful source for batch applications as well. It could 
> implement a bounded source. The primary question is how the bounds are 
> specified.
> One option : Source specifies a time period (say 9am-10am), and KafkaIO 
> fetches appropriate start and end offsets based on time-index in Kafka. This 
> would suite many batch applications that are launched on a scheduled.
> Another option is to always read till the end and commit the offsets to 
> Kafka. Handling failures and multiple runs of a task might be complicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2185) KafkaIO bounded source

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005711#comment-16005711
 ] 

Raghu Angadi commented on BEAM-2185:


Some more considerations : 
  - Splits:
 - Each partition should be split further. 
 - If we could fetch the size in bytes, it is easier to split. I don't 
think it is feasible get byte offsets.
 - Split could be based on message offsets, but the size could be too large 
or too small depending on average message size.
 - Take a hint from the user if the size is not known? May be not. Read a 
few sample records? Probably an over kill. 
  - It should support liquid-sharding (dynamic splitting). This does not need 
average message size. Just the offsets are good enough.
  

> KafkaIO bounded source
> --
>
> Key: BEAM-2185
> URL: https://issues.apache.org/jira/browse/BEAM-2185
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>
> KafkaIO could be a useful source for batch applications as well. It could 
> implement a bounded source. The primary question is how the bounds are 
> specified.
> One option : Source specifies a time period (say 9am-10am), and KafkaIO 
> fetches appropriate start and end offsets based on time-index in Kafka. This 
> would suite many batch applications that are launched on a scheduled.
> Another option is to always read till the end and commit the offsets to 
> Kafka. Handling failures and multiple runs of a task might be complicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2185) KafkaIO bounded source

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2185:
---
Description: 
KafkaIO could be a useful source for batch applications as well. It could 
implement a bounded source. The primary question is how the bounds are 
specified.

One option : Source specifies a time period (say 9am-10am), and KafkaIO fetches 
appropriate start and end offsets based on time-index in Kafka. This would 
suite many batch applications that are launched on a scheduled.

Another option is to always read till the end and commit the offsets to Kafka. 
Handling failures and multiple runs of a task might be complicated.



  was:
KafkaIO could be a useful source for batch applications as well. It could 
implement a bounded source. The primary question is how the bounds are 
specified. 
One option : source specifies a time period (say 9am-10am), and KafkaIO fetch 
appropriate start and end offsets based on time-index in Kafka. This would 
suite many batch applications that are lauched on a scheduled.

Another option is to always read till the end and commit the offsets to Kafka. 
Handling failures and multiple runs of a task might be complicated.




> KafkaIO bounded source
> --
>
> Key: BEAM-2185
> URL: https://issues.apache.org/jira/browse/BEAM-2185
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>
> KafkaIO could be a useful source for batch applications as well. It could 
> implement a bounded source. The primary question is how the bounds are 
> specified.
> One option : Source specifies a time period (say 9am-10am), and KafkaIO 
> fetches appropriate start and end offsets based on time-index in Kafka. This 
> would suite many batch applications that are launched on a scheduled.
> Another option is to always read till the end and commit the offsets to 
> Kafka. Handling failures and multiple runs of a task might be complicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2244) Move runner-facing Metrics classes to runners core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005703#comment-16005703
 ] 

ASF GitHub Bot commented on BEAM-2244:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3053


> Move runner-facing Metrics classes to runners core
> --
>
> Key: BEAM-2244
> URL: https://issues.apache.org/jira/browse/BEAM-2244
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2244) Move runner-facing Metrics classes to runners core

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-2244.
-
Resolution: Fixed

> Move runner-facing Metrics classes to runners core
> --
>
> Key: BEAM-2244
> URL: https://issues.apache.org/jira/browse/BEAM-2244
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3053: [BEAM-2244] Move details of metrics to runners core

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3053


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

http://git-wip-us.apache.org/repos/asf/beam/blob/8cd98bd9/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index 3dd98e0..9e71300 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -38,10 +38,10 @@ import static org.mockito.Mockito.withSettings;
 
 import com.google.common.collect.Iterables;
 import java.util.List;
+import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
 import org.apache.beam.runners.core.triggers.TriggerStateMachine;
 import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.sdk.metrics.MetricsContainer;
 import org.apache.beam.sdk.metrics.MetricsEnvironment;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -145,7 +145,8 @@ public class ReduceFnRunnerTest {
   @Test
   public void testOnElementBufferingDiscarding() throws Exception {
 // Test basic execution of a trigger using a non-combining window set and 
discarding mode.
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 ReduceFnTester tester =
 ReduceFnTester.nonCombining(FixedWindows.of(Duration.millis(10)), 
mockTriggerStateMachine,
 AccumulationMode.DISCARDING_FIRED_PANES, Duration.millis(100),
@@ -170,8 +171,7 @@ public class ReduceFnRunnerTest {
 // This element shouldn't be seen, because the trigger has finished
 injectElement(tester, 4);
 
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
-MetricName.named(ReduceFnRunner.class,
+long droppedElements = 
container.getCounter(MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
 assertEquals(1, droppedElements);
@@ -423,7 +423,8 @@ public class ReduceFnRunnerTest {
 
   @Test
   public void testWatermarkHoldAndLateData() throws Exception {
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 // Test handling of late data. Specifically, ensure the watermark hold is 
correct.
 ReduceFnTester tester =
 ReduceFnTester.nonCombining(FixedWindows.of(Duration.millis(10)), 
mockTriggerStateMachine,
@@ -452,7 +453,7 @@ public class ReduceFnRunnerTest {
 // Holding for the end-of-window transition.
 assertEquals(new Instant(9), tester.getWatermarkHold());
 // Nothing dropped.
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+long droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -514,7 +515,7 @@ public class ReduceFnRunnerTest {
 // Because we're about to expire the window, we output it.
 
when(mockTriggerStateMachine.shouldFire(anyTriggerContext())).thenReturn(false);
 injectElement(tester, 8);
-droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -1083,7 +1084,8 @@ public class ReduceFnRunnerTest {
*/
   @Test
   public void testDropDataMultipleWindowsFinishedTrigger() throws Exception {
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 ReduceFnTester tester = 
ReduceFnTester.combining(
 WindowingStrategy.of(
 SlidingWindows.of(Duration.millis(100)).every(Duration.millis(30)))
@@ -1097,7 +1099,7 @@ public class ReduceFnRunnerTest {
 // assigned to [-30, 70), [0, 100), [30, 130)
 TimestampedValue.of(12, new Instant(40)));
 
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+long droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -1109,7 +,7 @@ public class 

[5/5] beam git commit: This closes #3053

[dhalperi]

This closes #3053


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a39960b1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a39960b1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a39960b1

Branch: refs/heads/master
Commit: a39960b131674e6c1d120a971b91b37226c88414
Parents: 03a7f92 8cd98bd
Author: Dan Halperin 
Authored: Wed May 10 17:12:15 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 17:12:15 2017 -0700

--
 .../runners/core/metrics/MetricFiltering.java   | 102 
 .../beam/runners/core/metrics/MetricKey.java|  43 ++
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/metrics/MetricFilteringTest.java   | 148 ++
 .../apache/beam/runners/core/LateDataUtils.java |   4 +-
 .../beam/runners/core/metrics/CounterCell.java  |  86 
 .../beam/runners/core/metrics/DirtyState.java   |  99 
 .../runners/core/metrics/DistributionCell.java  |  80 +++
 .../runners/core/metrics/DistributionData.java  |  62 +++
 .../beam/runners/core/metrics/GaugeCell.java|  78 +++
 .../beam/runners/core/metrics/GaugeData.java|  83 
 .../beam/runners/core/metrics/MetricCell.java   |  42 ++
 .../runners/core/metrics/MetricUpdates.java |  79 +++
 .../core/metrics/MetricsContainerImpl.java  | 188 +++
 .../core/metrics/MetricsContainerStepMap.java   | 495 +++
 .../beam/runners/core/metrics/MetricsMap.java   |  88 
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/LateDataDroppingDoFnRunnerTest.java|   9 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |  34 +-
 .../runners/core/StatefulDoFnRunnerTest.java|  10 +-
 .../runners/core/metrics/CounterCellTest.java   |  56 +++
 .../runners/core/metrics/DirtyStateTest.java|  56 +++
 .../core/metrics/DistributionCellTest.java  |  54 ++
 .../runners/core/metrics/GaugeCellTest.java |  51 ++
 .../core/metrics/MetricUpdateMatchers.java  |  82 +++
 .../core/metrics/MetricsContainerImplTest.java  | 130 +
 .../metrics/MetricsContainerStepMapTest.java| 272 ++
 .../runners/core/metrics/MetricsMapTest.java| 103 
 .../beam/runners/direct/DirectMetrics.java  |  14 +-
 .../runners/direct/StepTransformResult.java |   2 +-
 .../beam/runners/direct/TransformExecutor.java  |  10 +-
 .../beam/runners/direct/TransformResult.java|   2 +-
 .../beam/runners/direct/DirectMetricsTest.java  |  14 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../metrics/DoFnRunnerWithMetricsUpdate.java|   3 +-
 .../flink/metrics/FlinkMetricContainer.java |   7 +-
 .../flink/metrics/MetricsAccumulator.java   |   2 +-
 .../flink/metrics/ReaderInvocationUtil.java |   5 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../beam/runners/dataflow/DataflowMetrics.java  |   4 +-
 .../runners/dataflow/DataflowMetricsTest.java   |   4 +-
 .../beam/runners/spark/SparkPipelineResult.java |   2 +-
 .../apache/beam/runners/spark/io/SourceRDD.java |   2 +-
 .../runners/spark/io/SparkUnboundedSource.java  |   2 +-
 .../spark/metrics/MetricsAccumulator.java   |   2 +-
 .../spark/metrics/MetricsAccumulatorParam.java  |   2 +-
 .../runners/spark/metrics/SparkBeamMetric.java  |   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |  10 +-
 .../spark/stateful/StateSpecFunctions.java  |   2 +-
 .../translation/DoFnRunnerWithMetrics.java  |   5 +-
 .../spark/translation/MultiDoFnFunction.java|   2 +-
 .../spark/translation/TransformTranslator.java  |   2 +-
 .../streaming/StreamingTransformTranslator.java |   2 +-
 .../spark/io/ReaderToIteratorAdapterTest.java   |   4 +-
 .../ResumeFromCheckpointStreamingTest.java  |   2 +-
 .../streaming/StreamingSourceMetricsTest.java   |   2 +-
 .../apache/beam/sdk/metrics/CounterCell.java|  72 ---
 .../org/apache/beam/sdk/metrics/DirtyState.java |  99 
 .../beam/sdk/metrics/DistributionCell.java  |  74 ---
 .../beam/sdk/metrics/DistributionData.java  |  60 ---
 .../beam/sdk/metrics/DistributionResult.java|   3 +
 .../org/apache/beam/sdk/metrics/GaugeCell.java  |  70 ---
 .../org/apache/beam/sdk/metrics/GaugeData.java  |  81 ---
 .../apache/beam/sdk/metrics/GaugeResult.java|   3 +
 .../org/apache/beam/sdk/metrics/Metric.java |   7 +-
 .../org/apache/beam/sdk/metrics/MetricCell.java |  53 --
 .../beam/sdk/metrics/MetricFiltering.java   |  99 
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  41 --
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  78 ---
 .../org/apache/beam/sdk/metrics/Metrics.java|   8 +-
 .../beam/sdk/metrics/MetricsContainer.java  | 146 +-
 .../sdk/metrics/MetricsContainerStepMap.java| 487 --
 .../beam/sdk/metrics/MetricsEnvironment.java|   9 +-
 .../org/apache/beam/sdk/metrics/MetricsMap.java |  87 
 

[1/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 03a7f92e3 -> a39960b13


http://git-wip-us.apache.org/repos/asf/beam/blob/8cd98bd9/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
deleted file mode 100644
index a0dd119..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
+++ /dev/null
@@ -1,242 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.metrics;
-
-import java.util.Objects;
-import org.apache.beam.sdk.metrics.MetricUpdates.MetricUpdate;
-import org.hamcrest.Description;
-import org.hamcrest.Matcher;
-import org.hamcrest.TypeSafeMatcher;
-
-/**
- * Matchers for metrics.
- */
-public class MetricMatchers {
-
-  /**
-   * Matches a {@link MetricUpdate} with the given name and contents.
-   *
-   * Visible since it may be used in runner-specific tests.
-   */
-  public static  Matcher metricUpdate(final String name, 
final T update) {
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean matchesSafely(MetricUpdate item) {
-return Objects.equals(name, item.getKey().metricName().name())
-&& Objects.equals(update, item.getUpdate());
-  }
-
-  @Override
-  public void describeTo(Description description) {
-description
-.appendText("MetricUpdate{name=").appendValue(name)
-.appendText(", update=").appendValue(update)
-.appendText("}");
-  }
-};
-  }
-
-  /**
-   * Matches a {@link MetricUpdate} with the given namespace, name, step and 
contents.
-   *
-   * Visible since it may be used in runner-specific tests.
-   */
-  public static  Matcher metricUpdate(
-  final String namespace, final String name, final String step, final T 
update) {
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean matchesSafely(MetricUpdate item) {
-return Objects.equals(namespace, 
item.getKey().metricName().namespace())
-&& Objects.equals(name, item.getKey().metricName().name())
-&& Objects.equals(step, item.getKey().stepName())
-&& Objects.equals(update, item.getUpdate());
-  }
-
-  @Override
-  public void describeTo(Description description) {
-description
-.appendText("MetricUpdate{inNamespace=").appendValue(namespace)
-.appendText(", name=").appendValue(name)
-.appendText(", step=").appendValue(step)
-.appendText(", update=").appendValue(update)
-.appendText("}");
-  }
-};
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for attempted metrics.
-   */
-  public static  Matcher attemptedMetricsResult(
-  final String namespace, final String name, final String step, final T 
value) {
-return metricsResult(namespace, name, step, value, false);
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for committed metrics.
-   */
-  public static  Matcher committedMetricsResult(
-  final String namespace, final String name, final String step, final T 
value) {
-return metricsResult(namespace, name, step, value, true);
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for either committed or attempted (based on {@code 
isCommitted}) metrics.
-   */
-  public static  Matcher metricsResult(
-  final String namespace, final String name, final String step, final T 
value,
-  final boolean isCommitted) {
-final String metricState = isCommitted ? "committed" : "attempted";
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean matchesSafely(MetricResult 

[4/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8cd98bd9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8cd98bd9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8cd98bd9

Branch: refs/heads/master
Commit: 8cd98bd968740e6f37e8ec9cf2d273c9f948698b
Parents: 03a7f92
Author: bchambers 
Authored: Tue May 9 15:45:50 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 17:12:12 2017 -0700

--
 .../runners/core/metrics/MetricFiltering.java   | 102 
 .../beam/runners/core/metrics/MetricKey.java|  43 ++
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/metrics/MetricFilteringTest.java   | 148 ++
 .../apache/beam/runners/core/LateDataUtils.java |   4 +-
 .../beam/runners/core/metrics/CounterCell.java  |  86 
 .../beam/runners/core/metrics/DirtyState.java   |  99 
 .../runners/core/metrics/DistributionCell.java  |  80 +++
 .../runners/core/metrics/DistributionData.java  |  62 +++
 .../beam/runners/core/metrics/GaugeCell.java|  78 +++
 .../beam/runners/core/metrics/GaugeData.java|  83 
 .../beam/runners/core/metrics/MetricCell.java   |  42 ++
 .../runners/core/metrics/MetricUpdates.java |  79 +++
 .../core/metrics/MetricsContainerImpl.java  | 188 +++
 .../core/metrics/MetricsContainerStepMap.java   | 495 +++
 .../beam/runners/core/metrics/MetricsMap.java   |  88 
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/LateDataDroppingDoFnRunnerTest.java|   9 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |  34 +-
 .../runners/core/StatefulDoFnRunnerTest.java|  10 +-
 .../runners/core/metrics/CounterCellTest.java   |  56 +++
 .../runners/core/metrics/DirtyStateTest.java|  56 +++
 .../core/metrics/DistributionCellTest.java  |  54 ++
 .../runners/core/metrics/GaugeCellTest.java |  51 ++
 .../core/metrics/MetricUpdateMatchers.java  |  82 +++
 .../core/metrics/MetricsContainerImplTest.java  | 130 +
 .../metrics/MetricsContainerStepMapTest.java| 272 ++
 .../runners/core/metrics/MetricsMapTest.java| 103 
 .../beam/runners/direct/DirectMetrics.java  |  14 +-
 .../runners/direct/StepTransformResult.java |   2 +-
 .../beam/runners/direct/TransformExecutor.java  |  10 +-
 .../beam/runners/direct/TransformResult.java|   2 +-
 .../beam/runners/direct/DirectMetricsTest.java  |  14 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../metrics/DoFnRunnerWithMetricsUpdate.java|   3 +-
 .../flink/metrics/FlinkMetricContainer.java |   7 +-
 .../flink/metrics/MetricsAccumulator.java   |   2 +-
 .../flink/metrics/ReaderInvocationUtil.java |   5 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../beam/runners/dataflow/DataflowMetrics.java  |   4 +-
 .../runners/dataflow/DataflowMetricsTest.java   |   4 +-
 .../beam/runners/spark/SparkPipelineResult.java |   2 +-
 .../apache/beam/runners/spark/io/SourceRDD.java |   2 +-
 .../runners/spark/io/SparkUnboundedSource.java  |   2 +-
 .../spark/metrics/MetricsAccumulator.java   |   2 +-
 .../spark/metrics/MetricsAccumulatorParam.java  |   2 +-
 .../runners/spark/metrics/SparkBeamMetric.java  |   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |  10 +-
 .../spark/stateful/StateSpecFunctions.java  |   2 +-
 .../translation/DoFnRunnerWithMetrics.java  |   5 +-
 .../spark/translation/MultiDoFnFunction.java|   2 +-
 .../spark/translation/TransformTranslator.java  |   2 +-
 .../streaming/StreamingTransformTranslator.java |   2 +-
 .../spark/io/ReaderToIteratorAdapterTest.java   |   4 +-
 .../ResumeFromCheckpointStreamingTest.java  |   2 +-
 .../streaming/StreamingSourceMetricsTest.java   |   2 +-
 .../apache/beam/sdk/metrics/CounterCell.java|  72 ---
 .../org/apache/beam/sdk/metrics/DirtyState.java |  99 
 .../beam/sdk/metrics/DistributionCell.java  |  74 ---
 .../beam/sdk/metrics/DistributionData.java  |  60 ---
 .../beam/sdk/metrics/DistributionResult.java|   3 +
 .../org/apache/beam/sdk/metrics/GaugeCell.java  |  70 ---
 .../org/apache/beam/sdk/metrics/GaugeData.java  |  81 ---
 .../apache/beam/sdk/metrics/GaugeResult.java|   3 +
 .../org/apache/beam/sdk/metrics/Metric.java |   7 +-
 .../org/apache/beam/sdk/metrics/MetricCell.java |  53 --
 .../beam/sdk/metrics/MetricFiltering.java   |  99 
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  41 --
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  78 ---
 .../org/apache/beam/sdk/metrics/Metrics.java| 

[2/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

http://git-wip-us.apache.org/repos/asf/beam/blob/8cd98bd9/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
deleted file mode 100644
index 4b8548f..000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.metrics;
-
-import java.util.concurrent.atomic.AtomicLong;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
-
-/**
- * Tracks the current value (and delta) for a Counter metric for a specific 
context and bundle.
- *
- * This class generally shouldn't be used directly. The only exception is 
within a runner where
- * a counter is being reported for a specific step (rather than the counter in 
the current context).
- * In that case retrieving the underlying cell and reporting directly to it 
avoids a step of
- * indirection.
- */
-@Experimental(Kind.METRICS)
-public class CounterCell implements MetricCell {
-
-  private final DirtyState dirty = new DirtyState();
-  private final AtomicLong value = new AtomicLong();
-
-  /**
-   * Package-visibility because all {@link CounterCell CounterCells} should be 
created by
-   * {@link MetricsContainer#getCounter(MetricName)}.
-   */
-  CounterCell() {}
-
-  /**
-   * Increment the counter by the given amount.
-   * @param n value to increment by. Can be negative to decrement.
-   */
-  public void update(long n) {
-value.addAndGet(n);
-dirty.afterModification();
-  }
-
-  @Override
-  public void update(Long n) {
-throw new UnsupportedOperationException("CounterCell.update(Long n) should 
not be used"
-+ " as it performs unnecessary boxing/unboxing. Use 
CounterCell.update(long n) instead.");
-  }
-
-  @Override public void update(MetricCell other) {
-update((long) other.getCumulative());
-  }
-
-  @Override
-  public DirtyState getDirty() {
-return dirty;
-  }
-
-  @Override
-  public Long getCumulative() {
-return value.get();
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/8cd98bd9/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
deleted file mode 100644
index 4e0c15c..000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.metrics;
-
-import java.io.Serializable;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
-
-/**
- * Atomically tracks the dirty-state of a metric.
- *
- * Reporting an update is split into two parts such that only changes made 
before the call to
- * {@link #beforeCommit()} are committed when {@link #afterCommit()} is 
invoked. This allows for
- * a two-step commit process of gathering all the 

[jira] [Commented] (BEAM-2244) Move runner-facing Metrics classes to runners core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005689#comment-16005689
 ] 

ASF GitHub Bot commented on BEAM-2244:
--

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/3064


> Move runner-facing Metrics classes to runners core
> --
>
> Key: BEAM-2244
> URL: https://issues.apache.org/jira/browse/BEAM-2244
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3064: [BEAM-2244] Cherry-pick #3053 to release-2.0.0

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/3064


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0ce5591c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0ce5591c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0ce5591c

Branch: refs/heads/release-2.0.0
Commit: 0ce5591c85be9d275082095041c55c26143109e7
Parents: e08cac0
Author: bchambers 
Authored: Tue May 9 15:45:50 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 17:01:32 2017 -0700

--
 .../runners/core/metrics/MetricFiltering.java   | 102 
 .../beam/runners/core/metrics/MetricKey.java|  43 ++
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/metrics/MetricFilteringTest.java   | 148 ++
 .../apache/beam/runners/core/LateDataUtils.java |   4 +-
 .../beam/runners/core/metrics/CounterCell.java  |  86 
 .../beam/runners/core/metrics/DirtyState.java   |  99 
 .../runners/core/metrics/DistributionCell.java  |  80 +++
 .../runners/core/metrics/DistributionData.java  |  62 +++
 .../beam/runners/core/metrics/GaugeCell.java|  78 +++
 .../beam/runners/core/metrics/GaugeData.java|  83 
 .../beam/runners/core/metrics/MetricCell.java   |  42 ++
 .../runners/core/metrics/MetricUpdates.java |  79 +++
 .../core/metrics/MetricsContainerImpl.java  | 188 +++
 .../core/metrics/MetricsContainerStepMap.java   | 495 +++
 .../beam/runners/core/metrics/MetricsMap.java   |  88 
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/LateDataDroppingDoFnRunnerTest.java|   9 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |  34 +-
 .../runners/core/StatefulDoFnRunnerTest.java|  10 +-
 .../runners/core/metrics/CounterCellTest.java   |  56 +++
 .../runners/core/metrics/DirtyStateTest.java|  56 +++
 .../core/metrics/DistributionCellTest.java  |  54 ++
 .../runners/core/metrics/GaugeCellTest.java |  51 ++
 .../core/metrics/MetricUpdateMatchers.java  |  82 +++
 .../core/metrics/MetricsContainerImplTest.java  | 130 +
 .../metrics/MetricsContainerStepMapTest.java| 272 ++
 .../runners/core/metrics/MetricsMapTest.java| 103 
 .../beam/runners/direct/DirectMetrics.java  |  14 +-
 .../runners/direct/StepTransformResult.java |   2 +-
 .../beam/runners/direct/TransformExecutor.java  |  10 +-
 .../beam/runners/direct/TransformResult.java|   2 +-
 .../beam/runners/direct/DirectMetricsTest.java  |  14 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../metrics/DoFnRunnerWithMetricsUpdate.java|   3 +-
 .../flink/metrics/FlinkMetricContainer.java |   7 +-
 .../flink/metrics/MetricsAccumulator.java   |   2 +-
 .../flink/metrics/ReaderInvocationUtil.java |   5 +-
 .../beam/runners/dataflow/DataflowMetrics.java  |   4 +-
 .../runners/dataflow/DataflowMetricsTest.java   |   4 +-
 .../beam/runners/spark/SparkPipelineResult.java |   2 +-
 .../apache/beam/runners/spark/io/SourceRDD.java |   2 +-
 .../runners/spark/io/SparkUnboundedSource.java  |   2 +-
 .../spark/metrics/MetricsAccumulator.java   |   2 +-
 .../spark/metrics/MetricsAccumulatorParam.java  |   2 +-
 .../runners/spark/metrics/SparkBeamMetric.java  |   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |  10 +-
 .../spark/stateful/StateSpecFunctions.java  |   2 +-
 .../translation/DoFnRunnerWithMetrics.java  |   5 +-
 .../spark/translation/MultiDoFnFunction.java|   2 +-
 .../spark/translation/TransformTranslator.java  |   2 +-
 .../streaming/StreamingTransformTranslator.java |   2 +-
 .../ResumeFromCheckpointStreamingTest.java  |   2 +-
 .../streaming/StreamingSourceMetricsTest.java   |   2 +-
 .../apache/beam/sdk/metrics/CounterCell.java|  72 ---
 .../org/apache/beam/sdk/metrics/DirtyState.java |  99 
 .../beam/sdk/metrics/DistributionCell.java  |  74 ---
 .../beam/sdk/metrics/DistributionData.java  |  60 ---
 .../beam/sdk/metrics/DistributionResult.java|   3 +
 .../org/apache/beam/sdk/metrics/GaugeCell.java  |  70 ---
 .../org/apache/beam/sdk/metrics/GaugeData.java  |  81 ---
 .../apache/beam/sdk/metrics/GaugeResult.java|   3 +
 .../org/apache/beam/sdk/metrics/Metric.java |   7 +-
 .../org/apache/beam/sdk/metrics/MetricCell.java |  53 --
 .../beam/sdk/metrics/MetricFiltering.java   |  99 
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  41 --
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  78 ---
 .../org/apache/beam/sdk/metrics/Metrics.java|   8 +-
 .../beam/sdk/metrics/MetricsContainer.java  | 146 +-
 

[2/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

http://git-wip-us.apache.org/repos/asf/beam/blob/0ce5591c/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
deleted file mode 100644
index 4e0c15c..000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DirtyState.java
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.metrics;
-
-import java.io.Serializable;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
-
-/**
- * Atomically tracks the dirty-state of a metric.
- *
- * Reporting an update is split into two parts such that only changes made 
before the call to
- * {@link #beforeCommit()} are committed when {@link #afterCommit()} is 
invoked. This allows for
- * a two-step commit process of gathering all the dirty updates (calling 
{#link beforeCommit()})
- * followed by committing and calling {#link afterCommit()}.
- *
- * The tracking of dirty states is done conservatively -- sometimes {@link 
#beforeCommit()}
- * will return true (indicating a dirty metric) even if there have been no 
changes since the last
- * commit.
- *
- * There is also a possible race when the underlying metric is modified but 
the call to
- * {@link #afterModification()} hasn't happened before the call to {@link 
#beforeCommit()}. In this
- * case the next round of metric updating will see the changes. If this was 
for the final commit,
- * then the metric updates shouldn't be extracted until all possible user 
modifications have
- * completed.
- */
-@Experimental(Kind.METRICS)
-class DirtyState implements Serializable {
-  private enum State {
-/** Indicates that there have been changes to the MetricCell since last 
commit. */
-DIRTY,
-/** Indicates that there have been no changes to the MetricCell since last 
commit. */
-CLEAN,
-/** Indicates that a commit of the current value is in progress. */
-COMMITTING
-  }
-
-  private final AtomicReference dirty = new 
AtomicReference<>(State.DIRTY);
-
-  /**
-   * Indicate that changes have been made to the metric being tracked by this 
{@link DirtyState}.
-   *
-   * Should be called after modification of the value.
-   */
-  public void afterModification() {
-dirty.set(State.DIRTY);
-  }
-
-  /**
-   * Check the dirty state and mark the metric as committing.
-   *
-   * If the state was {@code CLEAN}, this returns {@code false}. If the 
state was {@code DIRTY}
-   * or {@code COMMITTING} this returns {@code true} and sets the state to 
{@code COMMITTING}.
-   *
-   * @return {@code false} if the state is clean and {@code true} otherwise.
-   */
-  public boolean beforeCommit() {
-// After this loop, we want the state to be either CLEAN or COMMITTING.
-// If the state was CLEAN, we don't need to do anything (and exit the loop 
early)
-// If the state was DIRTY, we will attempt to do a CAS(DIRTY, COMMITTING). 
This will only
-// fail if another thread is getting updates which generally shouldn't be 
the case.
-// If the state was COMMITTING, we will attempt to do a CAS(COMMITTING, 
COMMITTING). This will
-// fail if another thread commits updates (which shouldn't be the case) or 
if the user code
-// updates the metric, in which case it will transition to DIRTY and the 
next iteration will
-// successfully update it.
-State state;
-do {
-  state = dirty.get();
-} while (state != State.CLEAN && !dirty.compareAndSet(state, 
State.COMMITTING));
-
-return state != State.CLEAN;
-  }
-
-  /**
-   * Mark any changes up to the most recently call to {@link #beforeCommit()}} 
as committed.
-   * The next call to {@link #beforeCommit()} will return {@code false} unless 
there have
-   * been changes made since the previous call to {@link #beforeCommit()}.
-   */
-  public void afterCommit() {
-dirty.compareAndSet(State.COMMITTING, State.CLEAN);
-  }
-}

[5/5] beam git commit: This closes #3064

[dhalperi]

This closes #3064


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7485583a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7485583a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7485583a

Branch: refs/heads/release-2.0.0
Commit: 7485583a326f1da4ec58ef2d3a550de1a283c3c1
Parents: e08cac0 0ce5591
Author: Dan Halperin 
Authored: Wed May 10 17:01:35 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 10 17:01:35 2017 -0700

--
 .../runners/core/metrics/MetricFiltering.java   | 102 
 .../beam/runners/core/metrics/MetricKey.java|  43 ++
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/metrics/MetricFilteringTest.java   | 148 ++
 .../apache/beam/runners/core/LateDataUtils.java |   4 +-
 .../beam/runners/core/metrics/CounterCell.java  |  86 
 .../beam/runners/core/metrics/DirtyState.java   |  99 
 .../runners/core/metrics/DistributionCell.java  |  80 +++
 .../runners/core/metrics/DistributionData.java  |  62 +++
 .../beam/runners/core/metrics/GaugeCell.java|  78 +++
 .../beam/runners/core/metrics/GaugeData.java|  83 
 .../beam/runners/core/metrics/MetricCell.java   |  42 ++
 .../runners/core/metrics/MetricUpdates.java |  79 +++
 .../core/metrics/MetricsContainerImpl.java  | 188 +++
 .../core/metrics/MetricsContainerStepMap.java   | 495 +++
 .../beam/runners/core/metrics/MetricsMap.java   |  88 
 .../beam/runners/core/metrics/package-info.java |  22 +
 .../core/LateDataDroppingDoFnRunnerTest.java|   9 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |  34 +-
 .../runners/core/StatefulDoFnRunnerTest.java|  10 +-
 .../runners/core/metrics/CounterCellTest.java   |  56 +++
 .../runners/core/metrics/DirtyStateTest.java|  56 +++
 .../core/metrics/DistributionCellTest.java  |  54 ++
 .../runners/core/metrics/GaugeCellTest.java |  51 ++
 .../core/metrics/MetricUpdateMatchers.java  |  82 +++
 .../core/metrics/MetricsContainerImplTest.java  | 130 +
 .../metrics/MetricsContainerStepMapTest.java| 272 ++
 .../runners/core/metrics/MetricsMapTest.java| 103 
 .../beam/runners/direct/DirectMetrics.java  |  14 +-
 .../runners/direct/StepTransformResult.java |   2 +-
 .../beam/runners/direct/TransformExecutor.java  |  10 +-
 .../beam/runners/direct/TransformResult.java|   2 +-
 .../beam/runners/direct/DirectMetricsTest.java  |  14 +-
 .../beam/runners/flink/FlinkRunnerResult.java   |   4 +-
 .../metrics/DoFnRunnerWithMetricsUpdate.java|   3 +-
 .../flink/metrics/FlinkMetricContainer.java |   7 +-
 .../flink/metrics/MetricsAccumulator.java   |   2 +-
 .../flink/metrics/ReaderInvocationUtil.java |   5 +-
 .../beam/runners/dataflow/DataflowMetrics.java  |   4 +-
 .../runners/dataflow/DataflowMetricsTest.java   |   4 +-
 .../beam/runners/spark/SparkPipelineResult.java |   2 +-
 .../apache/beam/runners/spark/io/SourceRDD.java |   2 +-
 .../runners/spark/io/SparkUnboundedSource.java  |   2 +-
 .../spark/metrics/MetricsAccumulator.java   |   2 +-
 .../spark/metrics/MetricsAccumulatorParam.java  |   2 +-
 .../runners/spark/metrics/SparkBeamMetric.java  |   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |  10 +-
 .../spark/stateful/StateSpecFunctions.java  |   2 +-
 .../translation/DoFnRunnerWithMetrics.java  |   5 +-
 .../spark/translation/MultiDoFnFunction.java|   2 +-
 .../spark/translation/TransformTranslator.java  |   2 +-
 .../streaming/StreamingTransformTranslator.java |   2 +-
 .../ResumeFromCheckpointStreamingTest.java  |   2 +-
 .../streaming/StreamingSourceMetricsTest.java   |   2 +-
 .../apache/beam/sdk/metrics/CounterCell.java|  72 ---
 .../org/apache/beam/sdk/metrics/DirtyState.java |  99 
 .../beam/sdk/metrics/DistributionCell.java  |  74 ---
 .../beam/sdk/metrics/DistributionData.java  |  60 ---
 .../beam/sdk/metrics/DistributionResult.java|   3 +
 .../org/apache/beam/sdk/metrics/GaugeCell.java  |  70 ---
 .../org/apache/beam/sdk/metrics/GaugeData.java  |  81 ---
 .../apache/beam/sdk/metrics/GaugeResult.java|   3 +
 .../org/apache/beam/sdk/metrics/Metric.java |   7 +-
 .../org/apache/beam/sdk/metrics/MetricCell.java |  53 --
 .../beam/sdk/metrics/MetricFiltering.java   |  99 
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  41 --
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  78 ---
 .../org/apache/beam/sdk/metrics/Metrics.java|   8 +-
 .../beam/sdk/metrics/MetricsContainer.java  | 146 +-
 .../sdk/metrics/MetricsContainerStepMap.java| 487 --
 .../beam/sdk/metrics/MetricsEnvironment.java|   9 +-
 .../org/apache/beam/sdk/metrics/MetricsMap.java |  87 
 .../apache/beam/sdk/metrics/SinkMetrics.java|   4 +
 .../apache/beam/sdk/metrics/SourceMetrics.java  |   3 

[1/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 e08cac055 -> 7485583a3


http://git-wip-us.apache.org/repos/asf/beam/blob/0ce5591c/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
deleted file mode 100644
index a0dd119..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
+++ /dev/null
@@ -1,242 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.metrics;
-
-import java.util.Objects;
-import org.apache.beam.sdk.metrics.MetricUpdates.MetricUpdate;
-import org.hamcrest.Description;
-import org.hamcrest.Matcher;
-import org.hamcrest.TypeSafeMatcher;
-
-/**
- * Matchers for metrics.
- */
-public class MetricMatchers {
-
-  /**
-   * Matches a {@link MetricUpdate} with the given name and contents.
-   *
-   * Visible since it may be used in runner-specific tests.
-   */
-  public static  Matcher metricUpdate(final String name, 
final T update) {
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean matchesSafely(MetricUpdate item) {
-return Objects.equals(name, item.getKey().metricName().name())
-&& Objects.equals(update, item.getUpdate());
-  }
-
-  @Override
-  public void describeTo(Description description) {
-description
-.appendText("MetricUpdate{name=").appendValue(name)
-.appendText(", update=").appendValue(update)
-.appendText("}");
-  }
-};
-  }
-
-  /**
-   * Matches a {@link MetricUpdate} with the given namespace, name, step and 
contents.
-   *
-   * Visible since it may be used in runner-specific tests.
-   */
-  public static  Matcher metricUpdate(
-  final String namespace, final String name, final String step, final T 
update) {
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean matchesSafely(MetricUpdate item) {
-return Objects.equals(namespace, 
item.getKey().metricName().namespace())
-&& Objects.equals(name, item.getKey().metricName().name())
-&& Objects.equals(step, item.getKey().stepName())
-&& Objects.equals(update, item.getUpdate());
-  }
-
-  @Override
-  public void describeTo(Description description) {
-description
-.appendText("MetricUpdate{inNamespace=").appendValue(namespace)
-.appendText(", name=").appendValue(name)
-.appendText(", step=").appendValue(step)
-.appendText(", update=").appendValue(update)
-.appendText("}");
-  }
-};
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for attempted metrics.
-   */
-  public static  Matcher attemptedMetricsResult(
-  final String namespace, final String name, final String step, final T 
value) {
-return metricsResult(namespace, name, step, value, false);
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for committed metrics.
-   */
-  public static  Matcher committedMetricsResult(
-  final String namespace, final String name, final String step, final T 
value) {
-return metricsResult(namespace, name, step, value, true);
-  }
-
-  /**
-   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose value equals
-   * the given value for either committed or attempted (based on {@code 
isCommitted}) metrics.
-   */
-  public static  Matcher metricsResult(
-  final String namespace, final String name, final String step, final T 
value,
-  final boolean isCommitted) {
-final String metricState = isCommitted ? "committed" : "attempted";
-return new TypeSafeMatcher() {
-  @Override
-  protected boolean 

[3/5] beam git commit: [BEAM-2244] Move details of Metrics to Runners Core

[dhalperi]

http://git-wip-us.apache.org/repos/asf/beam/blob/0ce5591c/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index 3dd98e0..9e71300 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -38,10 +38,10 @@ import static org.mockito.Mockito.withSettings;
 
 import com.google.common.collect.Iterables;
 import java.util.List;
+import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
 import org.apache.beam.runners.core.triggers.TriggerStateMachine;
 import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.sdk.metrics.MetricsContainer;
 import org.apache.beam.sdk.metrics.MetricsEnvironment;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -145,7 +145,8 @@ public class ReduceFnRunnerTest {
   @Test
   public void testOnElementBufferingDiscarding() throws Exception {
 // Test basic execution of a trigger using a non-combining window set and 
discarding mode.
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 ReduceFnTester tester =
 ReduceFnTester.nonCombining(FixedWindows.of(Duration.millis(10)), 
mockTriggerStateMachine,
 AccumulationMode.DISCARDING_FIRED_PANES, Duration.millis(100),
@@ -170,8 +171,7 @@ public class ReduceFnRunnerTest {
 // This element shouldn't be seen, because the trigger has finished
 injectElement(tester, 4);
 
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
-MetricName.named(ReduceFnRunner.class,
+long droppedElements = 
container.getCounter(MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
 assertEquals(1, droppedElements);
@@ -423,7 +423,8 @@ public class ReduceFnRunnerTest {
 
   @Test
   public void testWatermarkHoldAndLateData() throws Exception {
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 // Test handling of late data. Specifically, ensure the watermark hold is 
correct.
 ReduceFnTester tester =
 ReduceFnTester.nonCombining(FixedWindows.of(Duration.millis(10)), 
mockTriggerStateMachine,
@@ -452,7 +453,7 @@ public class ReduceFnRunnerTest {
 // Holding for the end-of-window transition.
 assertEquals(new Instant(9), tester.getWatermarkHold());
 // Nothing dropped.
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+long droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -514,7 +515,7 @@ public class ReduceFnRunnerTest {
 // Because we're about to expire the window, we output it.
 
when(mockTriggerStateMachine.shouldFire(anyTriggerContext())).thenReturn(false);
 injectElement(tester, 8);
-droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -1083,7 +1084,8 @@ public class ReduceFnRunnerTest {
*/
   @Test
   public void testDropDataMultipleWindowsFinishedTrigger() throws Exception {
-MetricsEnvironment.setCurrentContainer(new MetricsContainer("any"));
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
 ReduceFnTester tester = 
ReduceFnTester.combining(
 WindowingStrategy.of(
 SlidingWindows.of(Duration.millis(100)).every(Duration.millis(30)))
@@ -1097,7 +1099,7 @@ public class ReduceFnRunnerTest {
 // assigned to [-30, 70), [0, 100), [30, 130)
 TimestampedValue.of(12, new Instant(40)));
 
-long droppedElements = MetricsEnvironment.getCurrentContainer().getCounter(
+long droppedElements = container.getCounter(
 MetricName.named(ReduceFnRunner.class,
 ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
 .getCumulative().longValue();
@@ -1109,7 +,7 @@ public class 

[jira] [Commented] (BEAM-2185) KafkaIO bounded source

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005684#comment-16005684
 ] 

Xu Mingmin commented on BEAM-2185:
--

Good idea, I would prefer to option #1, that's the common case I use to ingest 
Kafka data into a batch process.

> KafkaIO bounded source
> --
>
> Key: BEAM-2185
> URL: https://issues.apache.org/jira/browse/BEAM-2185
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>
> KafkaIO could be a useful source for batch applications as well. It could 
> implement a bounded source. The primary question is how the bounds are 
> specified. 
> One option : source specifies a time period (say 9am-10am), and KafkaIO fetch 
> appropriate start and end offsets based on time-index in Kafka. This would 
> suite many batch applications that are lauched on a scheduled.
> Another option is to always read till the end and commit the offsets to 
> Kafka. Handling failures and multiple runs of a task might be complicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3069: Cherrypick #3068, updating mobile gaming javadoc, t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3069


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Correct javadoc for mobile gaming examples

[kenn]

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 24914cbf2 -> e08cac055


Correct javadoc for mobile gaming examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9fdd2e03
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9fdd2e03
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9fdd2e03

Branch: refs/heads/release-2.0.0
Commit: 9fdd2e03280abff883f8a1448aa2d1f243a42b2c
Parents: 24914cb
Author: Kenneth Knowles 
Authored: Wed May 10 16:41:00 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 16:45:52 2017 -0700

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  5 ++---
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 +-
 .../org/apache/beam/examples/complete/game/UserScore.java | 10 --
 4 files changed, 15 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9fdd2e03/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
index abbb13b..dbc7f88 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
@@ -78,17 +78,17 @@ import org.slf4j.LoggerFactory;
  * Run {@code org.apache.beam.examples.complete.game.injector.Injector} to 
generate
  * pubsub data for this pipeline. The {@code Injector} documentation provides 
more detail.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
- * like this:
+ * To execute this pipeline, specify the pipeline configuration like this:
  * {@code
  *   --project=YOUR_PROJECT_ID
  *   --tempLocation=gs://YOUR_TEMP_DIRECTORY
- *   --runner=BlockingDataflowRunner
+ *   --runner=YOUR_RUNNER
  *   --dataset=YOUR-DATASET
  *   --topic=projects/YOUR-PROJECT/topics/YOUR-TOPIC
  * }
  * 
- * where the BigQuery dataset you specify must already exist. The PubSub topic 
you specify should
+ *
+ * The BigQuery dataset you specify must already exist. The PubSub topic 
you specify should
  * be the same topic to which the Injector is publishing.
  */
 public class GameStats extends LeaderBoard {

http://git-wip-us.apache.org/repos/asf/beam/blob/9fdd2e03/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
index 6a322da..bab74ce 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
@@ -53,13 +53,12 @@ import org.joda.time.format.DateTimeFormatter;
  * {@link UserScore} pipeline. However, our batch processing is high-latency, 
in that we don't get
  * results from plays at the beginning of the batch's time period until the 
batch is processed.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
- * like this:
+ * To execute this pipeline, specify the pipeline configuration like this:
  * {@code
- *   --project=YOUR_PROJECT_ID
  *   --tempLocation=YOUR_TEMP_DIRECTORY
  *   --runner=YOUR_RUNNER
  *   --output=YOUR_OUTPUT_DIRECTORY
+ *   (possibly options specific to your runner or permissions for your 
temp/output locations)
  * }
  * 
  *

http://git-wip-us.apache.org/repos/asf/beam/blob/9fdd2e03/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
index f673a8d..b879b70 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
@@ -77,18 +77,18 @@ import org.joda.time.format.DateTimeFormatter;
  * Run {@code injector.Injector} to generate pubsub data for this pipeline. 
 The Injector
  * documentation provides more detail on how to do this.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline 

[2/2] beam git commit: This closes #3069: Cherrypick #3068, updating mobile gaming javadoc, to release-2.0.0

[kenn]

This closes #3069: Cherrypick #3068, updating mobile gaming javadoc, to 
release-2.0.0

  Correct javadoc for mobile gaming examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e08cac05
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e08cac05
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e08cac05

Branch: refs/heads/release-2.0.0
Commit: e08cac0550507cb883fa6f35c9fdfed63a25aaf4
Parents: 24914cb 9fdd2e0
Author: Kenneth Knowles 
Authored: Wed May 10 16:58:20 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 16:58:20 2017 -0700

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  5 ++---
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 +-
 .../org/apache/beam/examples/complete/game/UserScore.java | 10 --
 4 files changed, 15 insertions(+), 18 deletions(-)
--

[GitHub] beam pull request #3069: Cherrypick #3068, updating mobile gaming javadoc, t...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3069

Cherrypick #3068, updating mobile gaming javadoc, to release-2.0.0

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay 
CC: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam UserScore-2.0.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3069.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3069


commit 9fdd2e03280abff883f8a1448aa2d1f243a42b2c
Author: Kenneth Knowles 
Date:   2017-05-10T23:41:00Z

Correct javadoc for mobile gaming examples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3068: Correct javadoc for mobile gaming examples

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3068


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Correct javadoc for mobile gaming examples

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master c6ac3eca0 -> 03a7f92e3


Correct javadoc for mobile gaming examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/968a59f7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/968a59f7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/968a59f7

Branch: refs/heads/master
Commit: 968a59f7c77fd35757aad0c7869292c80684d21a
Parents: c6ac3ec
Author: Kenneth Knowles 
Authored: Wed May 10 16:41:00 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 16:42:07 2017 -0700

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  5 ++---
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 +-
 .../org/apache/beam/examples/complete/game/UserScore.java | 10 --
 4 files changed, 15 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/968a59f7/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
index abbb13b..dbc7f88 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
@@ -78,17 +78,17 @@ import org.slf4j.LoggerFactory;
  * Run {@code org.apache.beam.examples.complete.game.injector.Injector} to 
generate
  * pubsub data for this pipeline. The {@code Injector} documentation provides 
more detail.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
- * like this:
+ * To execute this pipeline, specify the pipeline configuration like this:
  * {@code
  *   --project=YOUR_PROJECT_ID
  *   --tempLocation=gs://YOUR_TEMP_DIRECTORY
- *   --runner=BlockingDataflowRunner
+ *   --runner=YOUR_RUNNER
  *   --dataset=YOUR-DATASET
  *   --topic=projects/YOUR-PROJECT/topics/YOUR-TOPIC
  * }
  * 
- * where the BigQuery dataset you specify must already exist. The PubSub topic 
you specify should
+ *
+ * The BigQuery dataset you specify must already exist. The PubSub topic 
you specify should
  * be the same topic to which the Injector is publishing.
  */
 public class GameStats extends LeaderBoard {

http://git-wip-us.apache.org/repos/asf/beam/blob/968a59f7/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
index 6a322da..bab74ce 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
@@ -53,13 +53,12 @@ import org.joda.time.format.DateTimeFormatter;
  * {@link UserScore} pipeline. However, our batch processing is high-latency, 
in that we don't get
  * results from plays at the beginning of the batch's time period until the 
batch is processed.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
- * like this:
+ * To execute this pipeline, specify the pipeline configuration like this:
  * {@code
- *   --project=YOUR_PROJECT_ID
  *   --tempLocation=YOUR_TEMP_DIRECTORY
  *   --runner=YOUR_RUNNER
  *   --output=YOUR_OUTPUT_DIRECTORY
+ *   (possibly options specific to your runner or permissions for your 
temp/output locations)
  * }
  * 
  *

http://git-wip-us.apache.org/repos/asf/beam/blob/968a59f7/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
index f673a8d..b879b70 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
@@ -77,18 +77,18 @@ import org.joda.time.format.DateTimeFormatter;
  * Run {@code injector.Injector} to generate pubsub data for this pipeline. 
 The Injector
  * documentation provides more detail on how to do this.
  *
- * To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
- * 

[2/2] beam git commit: This closes #3068: Correct javadoc for mobile gaming examples

[kenn]

This closes #3068: Correct javadoc for mobile gaming examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/03a7f92e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/03a7f92e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/03a7f92e

Branch: refs/heads/master
Commit: 03a7f92e3da02c7aad0271ddeed7f540bcaa0979
Parents: c6ac3ec 968a59f
Author: Kenneth Knowles 
Authored: Wed May 10 16:54:30 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 16:54:30 2017 -0700

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  5 ++---
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 +-
 .../org/apache/beam/examples/complete/game/UserScore.java | 10 --
 4 files changed, 15 insertions(+), 18 deletions(-)
--

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3761

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3068: Correct javadoc for mobile gaming examples

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3068

Correct javadoc for mobile gaming examples

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam UserScore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3068.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3068


commit 968a59f7c77fd35757aad0c7869292c80684d21a
Author: Kenneth Knowles 
Date:   2017-05-10T23:41:00Z

Correct javadoc for mobile gaming examples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2008) aggregation functions support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005666#comment-16005666
 ] 

ASF GitHub Bot commented on BEAM-2008:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3067

[BEAM-2008] aggregation functions support

Support common-used aggregation functions in SQL, including:  
`COUNT`, `SUM`, `AVG`, `MAX`, `MIN`

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2008

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3067


commit 9d918e0e0802e2b6618afdf7d162f911249050d4
Author: mingmxu 
Date:   2017-05-10T23:38:13Z

Support common-used aggregation functions in SQL, including:
  COUNT,SUM,AVG,MAX,MIN




> aggregation functions support
> -
>
> Key: BEAM-2008
> URL: https://issues.apache.org/jira/browse/BEAM-2008
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> Support common-used aggregation functions in SQL, including:
> COUNT
> SUM
> MAX
> MIN



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3067: [BEAM-2008] aggregation functions support

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3067

[BEAM-2008] aggregation functions support

Support common-used aggregation functions in SQL, including:  
`COUNT`, `SUM`, `AVG`, `MAX`, `MIN`

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2008

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3067


commit 9d918e0e0802e2b6618afdf7d162f911249050d4
Author: mingmxu 
Date:   2017-05-10T23:38:13Z

Support common-used aggregation functions in SQL, including:
  COUNT,SUM,AVG,MAX,MIN




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2776

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3066: Cherry-pick #3062: Skip generating empty jars for p...

[lukecwik]

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/3066


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Skip generating empty jars for parent poms

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 697427a40 -> 24914cbf2


Skip generating empty jars for parent poms


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/194b99f3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/194b99f3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/194b99f3

Branch: refs/heads/release-2.0.0
Commit: 194b99f3693e0a6f8dc9686cdd1f5d534300563b
Parents: 697427a
Author: Vikas Kedigehalli 
Authored: Wed May 10 14:29:45 2017 -0700
Committer: Luke Cwik 
Committed: Wed May 10 16:08:51 2017 -0700

--
 pom.xml | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/194b99f3/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 93fe29c..d192c8b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1195,6 +1195,9 @@
   
 
   
+  
+true
+  
 
 
 

[2/2] beam git commit: Cherry-pick #3062: Skip generating empty jars for parent poms

[lcwik]

Cherry-pick #3062: Skip generating empty jars for parent poms

This closes #3066


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/24914cbf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/24914cbf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/24914cbf

Branch: refs/heads/release-2.0.0
Commit: 24914cbf2737eb203e33a882b89aed5f148c88c0
Parents: 697427a 194b99f
Author: Luke Cwik 
Authored: Wed May 10 16:22:24 2017 -0700
Committer: Luke Cwik 
Committed: Wed May 10 16:22:24 2017 -0700

--
 pom.xml | 3 +++
 1 file changed, 3 insertions(+)
--

[2/2] beam git commit: Update SDK Coders to return the Empty List from getCoderArguments

[lcwik]

Update SDK Coders to return the Empty List from getCoderArguments

This closes #3045


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c6ac3eca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c6ac3eca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c6ac3eca

Branch: refs/heads/master
Commit: c6ac3eca06ef4ef85f9ca2feb3749d913151feef
Parents: 40d7306 5a4e131
Author: Luke Cwik 
Authored: Wed May 10 16:18:49 2017 -0700
Committer: Luke Cwik 
Committed: Wed May 10 16:18:49 2017 -0700

--
 .../main/java/org/apache/beam/sdk/coders/AvroCoder.java |  5 -
 .../java/org/apache/beam/sdk/coders/ByteArrayCoder.java |  6 --
 .../org/apache/beam/sdk/transforms/join/UnionCoder.java |  3 ++-
 .../java/org/apache/beam/sdk/util/WindowedValue.java| 12 ++--
 .../org/apache/beam/sdk/transforms/CombineFnsTest.java  |  3 ++-
 5 files changed, 14 insertions(+), 15 deletions(-)
--

[GitHub] beam pull request #3045: Update SDK Coders to return the Empty List from get...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3045


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Update SDK Coders to return the Empty List from getCoderArguments

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 40d73068d -> c6ac3eca0


Update SDK Coders to return the Empty List from getCoderArguments

Brings existing Coders in line with the documentation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5a4e1314
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5a4e1314
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5a4e1314

Branch: refs/heads/master
Commit: 5a4e13149fb9648cb19428e8eef1fdddf2169785
Parents: 40d7306
Author: Thomas Groh 
Authored: Wed May 10 08:56:54 2017 -0700
Committer: Luke Cwik 
Committed: Wed May 10 16:18:19 2017 -0700

--
 .../main/java/org/apache/beam/sdk/coders/AvroCoder.java |  5 -
 .../java/org/apache/beam/sdk/coders/ByteArrayCoder.java |  6 --
 .../org/apache/beam/sdk/transforms/join/UnionCoder.java |  3 ++-
 .../java/org/apache/beam/sdk/util/WindowedValue.java| 12 ++--
 .../org/apache/beam/sdk/transforms/CombineFnsTest.java  |  3 ++-
 5 files changed, 14 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5a4e1314/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
index bba669d..94b4a9f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
@@ -318,11 +318,6 @@ public class AvroCoder extends CustomCoder {
 return reader.get().read(null, decoderInstance);
   }
 
-  @Override
-  public List> getCoderArguments() {
-return null;
-  }
-
   /**
* @throws NonDeterministicException when the type may not be 
deterministically
* encoded using the given {@link Schema}, the {@code directBinaryEncoder}, 
and the

http://git-wip-us.apache.org/repos/asf/beam/blob/5a4e1314/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
index 3b38388..16094dd 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
@@ -21,7 +21,6 @@ import com.google.common.io.ByteStreams;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
-import java.util.List;
 import org.apache.beam.sdk.util.ExposedByteArrayOutputStream;
 import org.apache.beam.sdk.util.StreamUtils;
 import org.apache.beam.sdk.util.VarInt;
@@ -113,11 +112,6 @@ public class ByteArrayCoder extends AtomicCoder {
   }
 
   @Override
-  public List> getCoderArguments() {
-return null;
-  }
-
-  @Override
   public void verifyDeterministic() {}
 
   /**

http://git-wip-us.apache.org/repos/asf/beam/blob/5a4e1314/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/UnionCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/UnionCoder.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/UnionCoder.java
index 66959d3..6590ca0 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/UnionCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/UnionCoder.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.transforms.join;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.util.Collections;
 import java.util.List;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
@@ -94,7 +95,7 @@ public class UnionCoder extends 
StructuredCoder {
 
   @Override
   public List> getCoderArguments() {
-return null;
+return Collections.emptyList();
   }
 
   @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/5a4e1314/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
index 444521a..8300954 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
@@ -685,9 +685,17 @@ public abstract class WindowedValue 

[GitHub] beam pull request #3066: Cherry-pick #3063: Skip generating empty jars for p...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3066

Cherry-pick #3063: Skip generating empty jars for parent poms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam pr3063

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3066.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3066


commit 194b99f3693e0a6f8dc9686cdd1f5d534300563b
Author: Vikas Kedigehalli 
Date:   2017-05-10T21:29:45Z

Skip generating empty jars for parent poms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Skip generating empty jars for parent modules

[lcwik]

Skip generating empty jars for parent modules

This closes #3062


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/40d73068
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/40d73068
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/40d73068

Branch: refs/heads/master
Commit: 40d73068d950bc9387761bfaa8507b9e1dc507ee
Parents: 8994afa 2c7abac
Author: Luke Cwik 
Authored: Wed May 10 16:07:40 2017 -0700
Committer: Luke Cwik 
Committed: Wed May 10 16:07:40 2017 -0700

--
 pom.xml | 3 +++
 1 file changed, 3 insertions(+)
--

[1/2] beam git commit: Skip generating empty jars for parent poms

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 8994afa51 -> 40d73068d


Skip generating empty jars for parent poms


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2c7abaca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2c7abaca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2c7abaca

Branch: refs/heads/master
Commit: 2c7abaca1112e56290a7ecb886d59805df5f69ee
Parents: 8994afa
Author: Vikas Kedigehalli 
Authored: Wed May 10 14:29:45 2017 -0700
Committer: Vikas Kedigehalli 
Committed: Wed May 10 15:25:23 2017 -0700

--
 pom.xml | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2c7abaca/pom.xml
--
diff --git a/pom.xml b/pom.xml
index a97479c..e7eb729 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1195,6 +1195,9 @@
   
 
   
+  
+true
+  
 
 
 

[GitHub] beam pull request #3062: Skip generating empty jars for parent modules

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3062


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3065: Make a couple more things private in the Python SDK

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3065

Make a couple more things private in the Python SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam py-private

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3065.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3065


commit 7deb54fcd4f809b11f880440be88b82f5f93b0e4
Author: Robert Bradshaw 
Date:   2017-05-10T22:55:46Z

Remove some internal details from the public API.

commit 1532055ddfb97c9c56ff3bf3fb9837456a821c4a
Author: Robert Bradshaw 
Date:   2017-05-10T22:56:50Z

Make GroupByKeyOnly private.

commit f8f2993a86ca07240a958f738f2eb2c74883b4b2
Author: Robert Bradshaw 
Date:   2017-05-10T23:04:49Z

more clone changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2244) Move runner-facing Metrics classes to runners core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005601#comment-16005601
 ] 

ASF GitHub Bot commented on BEAM-2244:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/3064

[BEAM-2244] Cherry-pick #3053 to release-2.0.0

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.

R: @bjchambers confirm this is a valid CP. Also, note there was one merge 
conflict fixup in SourceRdd -- check that one out.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam cp-3053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3064


commit e2f331e517804cc3ed0262fe8cc0772a4629d310
Author: bchambers 
Date:   2017-05-09T22:45:50Z

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.




> Move runner-facing Metrics classes to runners core
> --
>
> Key: BEAM-2244
> URL: https://issues.apache.org/jira/browse/BEAM-2244
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Ben Chambers
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3064: [BEAM-2244] Cherry-pick #3053 to release-2.0.0

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/3064

[BEAM-2244] Cherry-pick #3053 to release-2.0.0

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.

R: @bjchambers confirm this is a valid CP. Also, note there was one merge 
conflict fixup in SourceRdd -- check that one out.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam cp-3053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3064


commit e2f331e517804cc3ed0262fe8cc0772a4629d310
Author: bchambers 
Date:   2017-05-09T22:45:50Z

[BEAM-2244] Move details of Metrics to Runners Core

Largeish changes this required were:

  - splitting the MetricsContainer into an interface in Java Core with
an implementation in Runners Core
  - modifying the various *Cell classes to have a name
  - cleaning up dependency cross-fire.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2775

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2223) java8 examples are not running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005559#comment-16005559
 ] 

ASF GitHub Bot commented on BEAM-2223:
--

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/2983


> java8 examples are not running
> --
>
> Key: BEAM-2223
> URL: https://issues.apache.org/jira/browse/BEAM-2223
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Ahmet Altay
>Assignee: Kenneth Knowles
> Fix For: 2.0.0
>
>
> Could not run java8 examples any more with:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.UserScore 
> -Dexec.args="--project= --dataset= 
> --tempLocation="
> {code}
> Fails with:
> {code}
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: org/hamcrest/Matcher
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.getDeclaredMethods0(Native Method)
>   at java.lang.Class.privateGetDeclaredMethods(Class.java:2703)
>   at java.lang.Class.privateGetPublicMethods(Class.java:2904)
>   at java.lang.Class.privateGetPublicMethods(Class.java:2913)
>   at java.lang.Class.getMethods(Class.java:1617)
>   at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:451)
>   at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:339)
>   at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:639)
>   at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557)
>   at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230)
>   at java.lang.reflect.WeakCache.get(WeakCache.java:127)
>   at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419)
>   at java.lang.reflect.Proxy.getProxyClass(Proxy.java:371)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:606)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:544)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:570)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:519)
>   at 
> org.apache.beam.examples.complete.game.UserScore.main(UserScore.java:226)
>   ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.hamcrest.Matcher
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 35 more
> {code}
> cc: [~kenn][~tgroh][~vikasrk][~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2983: [BEAM-2223] PipelineOptionsFactory: improve debugga...

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/2983


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3063: Fix a typo in TestDataflowRunnerTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3063


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fix a typo in TestDataflowRunnerTest

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master d32213bfc -> 8994afa51


Fix a typo in TestDataflowRunnerTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/22d87785
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/22d87785
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/22d87785

Branch: refs/heads/master
Commit: 22d87785cf1bb32af31628813f20a2d2ad854dcd
Parents: d32213b
Author: Vikas Kedigehalli 
Authored: Wed May 10 15:19:50 2017 -0700
Committer: Vikas Kedigehalli 
Committed: Wed May 10 15:19:50 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/22d87785/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
index 1c0876a..15faf26 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
@@ -562,7 +562,7 @@ public class TestDataflowRunnerTest {
* Tests that when a streaming pipeline terminates in FAIL that the {@link
* TestPipelineOptions#setOnSuccessMatcher(SerializableMatcher) on success 
matcher} is not
* invoked.
-   */t ad
+   */
   @Test
   public void testStreamingOnSuccessMatcherWhenPipelineFails() throws 
Exception {
 options.setStreaming(true);

[2/2] beam git commit: This closes #3063: Fix a typo in TestDataflowRunnerTest

[kenn]

This closes #3063: Fix a typo in TestDataflowRunnerTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8994afa5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8994afa5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8994afa5

Branch: refs/heads/master
Commit: 8994afa51d2cac77198b07bb06087019a3bf7836
Parents: d32213b 22d8778
Author: Kenneth Knowles 
Authored: Wed May 10 15:22:59 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 10 15:22:59 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #3063: Fix a typo in TestDataflowRunnerTest

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3063

Fix a typo in TestDataflowRunnerTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3063.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3063


commit 22d87785cf1bb32af31628813f20a2d2ad854dcd
Author: Vikas Kedigehalli 
Date:   2017-05-10T22:19:50Z

Fix a typo in TestDataflowRunnerTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3759

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2251) Create an examples/java and examples/java8 uber jar that doesn't contain a runner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2251:

Description: 
In addition to the regular artifact examples/java and examples/java8 that are 
produced, consider making an uber jar as an additional attached artifact that 
shades all the dependencies for easier command line usage so that it can be 
joined with a runner uber jar.
{code}
java -cp examples.jar:myrunner.jar WordCount ...
{code}

Also look at the maven archetypes found in:
https://github.com/apache/beam/tree/master/sdks/java/maven-archetypes

  was:
In addition to the regular artifact examples/java and examples/java8 that are 
produced, consider making an uber jar as an additional attached artifact that 
shades all the dependencies for easier command line usage so that it can be 
joined with a runner uber jar.
{code}
java -cp examples.jar:myrunner.jar WordCount ...
{code}

Also look at the maven archetypes that are generated.


> Create an examples/java and examples/java8 uber jar that doesn't contain a 
> runner
> -
>
> Key: BEAM-2251
> URL: https://issues.apache.org/jira/browse/BEAM-2251
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Luke Cwik
>Priority: Minor
>
> In addition to the regular artifact examples/java and examples/java8 that are 
> produced, consider making an uber jar as an additional attached artifact that 
> shades all the dependencies for easier command line usage so that it can be 
> joined with a runner uber jar.
> {code}
> java -cp examples.jar:myrunner.jar WordCount ...
> {code}
> Also look at the maven archetypes found in:
> https://github.com/apache/beam/tree/master/sdks/java/maven-archetypes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2251) Create an examples/java and examples/java8 uber jar that doesn't contain a runner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2251:

Description: 
In addition to the regular artifact examples/java and examples/java8 that are 
produced, consider making an uber jar as an additional attached artifact that 
shades all the dependencies for easier command line usage so that it can be 
joined with a runner uber jar.
{code}
java -cp examples.jar:myrunner.jar WordCount ...
{code}

Also look at the maven archetypes that are generated.

  was:
In addition to the regular artifact examples/java and examples/java8 that are 
produced, consider making an uber jar as an additional attached artifact that 
shades all the dependencies for easier command line usage so that it can be 
joined with a runner uber jar.
{code}
java -cp examples.jar:myrunner.jar WordCount ...
{code}


> Create an examples/java and examples/java8 uber jar that doesn't contain a 
> runner
> -
>
> Key: BEAM-2251
> URL: https://issues.apache.org/jira/browse/BEAM-2251
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Luke Cwik
>Priority: Minor
>
> In addition to the regular artifact examples/java and examples/java8 that are 
> produced, consider making an uber jar as an additional attached artifact that 
> shades all the dependencies for easier command line usage so that it can be 
> joined with a runner uber jar.
> {code}
> java -cp examples.jar:myrunner.jar WordCount ...
> {code}
> Also look at the maven archetypes that are generated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2253) maven archetype poms should have versions controlled automatically based upon root pom.xml versions

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2253:
---

 Summary: maven archetype poms should have versions controlled 
automatically based upon root pom.xml versions
 Key: BEAM-2253
 URL: https://issues.apache.org/jira/browse/BEAM-2253
 Project: Beam
  Issue Type: Improvement
  Components: examples-java
Affects Versions: 2.1.0
Reporter: Luke Cwik
Priority: Minor


Use a tool which sets the versions of plugins/dependencies during the build 
process within:
https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
from:
https://github.com/apache/beam/blob/master/pom.xml



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2252) maven-shade-plugin should be defined in pluginManagement instead of plugins so that the plugin execution order defined by the root pom.xml is used

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2252:
---

Assignee: (was: Daniel Halperin)

> maven-shade-plugin should be defined in pluginManagement instead of plugins 
> so that the plugin execution order defined by the root pom.xml is used
> --
>
> Key: BEAM-2252
> URL: https://issues.apache.org/jira/browse/BEAM-2252
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-direct, sdk-java-core, 
> sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Luke Cwik
>Priority: Minor
>
> Several of our child pom.xml declare maven-shade-plugin within build/plugins 
> instead of build/pluginManagement which affects the plugin execution order.
> See BEAM-2242 for when we surgically fixed how shading was performed and this 
> issue is about revisiting that change and improving upon it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3760

[Apache Jenkins Server]

See 


Changes:

[klk] Allow any throwable in PAssert to constitute adequate failure

[klk] TestDataflowRunner: throw AssertionError only when assertion known

[klk] Re-enable UsesTimersInParDo tests in Dataflow runner

--
[...truncated 879.43 KB...]
2017-05-10T21:33:44.912 [INFO] Exclude: **/test/**/.placeholder
2017-05-10T21:33:44.912 [INFO] Exclude: .repository/**/*
2017-05-10T21:33:44.912 [INFO] Exclude: **/nose-*.egg/**/*
2017-05-10T21:33:44.912 [INFO] Exclude: **/.eggs/**/*
2017-05-10T21:33:44.912 [INFO] Exclude: **/.tox/**/*
2017-05-10T21:33:44.913 [INFO] Exclude: **/.checkstyle
2017-05-10T21:33:44.913 [INFO] Exclude: **/.classpath
2017-05-10T21:33:44.913 [INFO] Exclude: **/.factorypath
2017-05-10T21:33:44.913 [INFO] Exclude: **/.project
2017-05-10T21:33:44.913 [INFO] Exclude: **/.settings/**/*
2017-05-10T21:33:44.913 [INFO] Exclude: 
**/sdks/python/apache_beam/runners/api/*.py
2017-05-10T21:33:44.916 [INFO] 9 resources included (use -debug for more 
details)
2017-05-10T21:33:44.939 [INFO] Rat check: Summary over all files. Unapproved: 
0, unknown: 0, generated: 0, approved: 9 licenses.
2017-05-10T21:33:45.554 [INFO] 
2017-05-10T21:33:45.554 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-sdks-java-io-hadoop-input-format ---
2017-05-10T21:33:45.555 [INFO] Installing 

 to 

2017-05-10T21:33:45.565 [INFO] Installing 

 to 

2017-05-10T21:33:45.566 [INFO] Installing 

 to 

2017-05-10T21:33:45.568 [INFO] Installing 

 to 

2017-05-10T21:33:45.572 [INFO] Installing 

 to 

2017-05-10T21:33:45.575 [INFO] Installing 

 to 

2017-05-10T21:33:45.577 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-05-10T21:33:51.212 [INFO]  
   
2017-05-10T21:33:51.212 [INFO] 

2017-05-10T21:33:51.212 [INFO] Building Apache Beam :: Runners :: Google Cloud 
Dataflow 2.1.0-SNAPSHOT
2017-05-10T21:33:51.212 [INFO] 

2017-05-10T21:33:51.224 [INFO] Downloading: 

[jira] [Created] (BEAM-2252) maven-shade-plugin should be defined in pluginManagement instead of plugins so that the plugin execution order defined by the root pom.xml is used

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2252:
---

 Summary: maven-shade-plugin should be defined in pluginManagement 
instead of plugins so that the plugin execution order defined by the root 
pom.xml is used
 Key: BEAM-2252
 URL: https://issues.apache.org/jira/browse/BEAM-2252
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow, runner-direct, sdk-java-core, 
sdk-java-gcp
Affects Versions: 2.1.0
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


Several of our child pom.xml declare maven-shade-plugin within build/plugins 
instead of build/pluginManagement which affects the plugin execution order.

See BEAM-2242 for when we surgically fixed how shading was performed and this 
issue is about revisiting that change and improving upon it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3062: Skip generating empty jars for parent poms

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3062

Skip generating empty jars for parent poms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam jar_plugin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3062.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3062


commit 96528716f72451baed1640981d7a2131d9087500
Author: Vikas Kedigehalli 
Date:   2017-05-10T21:29:45Z

Skip generating empty jars for parent poms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005465#comment-16005465
 ] 

Eugene Kirpichov edited comment on BEAM-2140 at 5/10/17 9:22 PM:
-

Aljoscha - SDF code does not inspect watermarks.
Here's what should happen really, when you apply an SDF to a BoundedSource that 
contains exactly 1 element (with more elements, it'll be more of the same).

1. We read an element from the source, and as it goes through the SDF 
expansion, it ends up in ProcessFn.
2. ProcessFn processes this element and its restriction, and if there's a 
residual restriction (checkpoint), then it sets a watermark hold and sets a 
timer to continue the processing.
3. The BoundedSource is done, so its watermark progresses to infinity - but 
this is fine. The input watermark of ProcessFn does NOT progress to infinity 
just yet, because it has set a watermark hold! (if it didn't set the hold, then 
its input watermark would also progress to infinity, and the timer would be 
late-data and hence dropped)
4. The timer set by ProcessFn fires, and it processes (calls ProcessElement) 
some more; again possibly setting a watermark hold and setting another timer to 
continue the processing. And so on.
5. Eventually the ProcessElement call finishes without producing a residual 
restriction. In that case, ProcessFn a) clears the watermark hold b) does NOT 
set a continuation timer.
6. After that, watermark of ProcessFn itself progresses to infinity (because 
there's no hold anymore) and the pipeline terminates.

I suspect that in the Flink implementation, something is going wrong between 
steps 3 and 4. E.g. maybe the watermark hold isn't working (i.e. isn't 
preventing the watermark of ProcessFn from progressing to infinity); or maybe 
somehow the processing-time timer gets dropped for a different reason.


was (Author: jkff):
Aljoscha - SDF code does not inspect watermarks.
Here's what should happen really, when you apply an SDF to a BoundedSource that 
contains exactly 1 element (with more elements, it'll be more of the same).

1. We read an element from the source, and as it goes through the SDF 
expansion, it ends up in ProcessFn.
2. ProcessFn processes this element and its restriction, and if there's a 
residual restriction (checkpoint), then it sets a watermark hold and sets a 
timer to continue the processing.
3. The BoundedSource is done, so its watermark progresses to infinity - but 
this is fine. The input watermark of ProcessFn does NOT progress to infinity 
just yet, because it has set a watermark hold! (if it didn't set the hold, then 
its input watermark would also progress to infinity, and the timer would be 
dropped)
4. The timer set by ProcessFn fires, and it processes (calls ProcessElement) 
some more; again possibly setting a watermark hold and setting another timer to 
continue the processing. And so on.
5. Eventually the ProcessElement call finishes without producing a residual 
restriction. In that case, ProcessFn a) clears the watermark hold b) does NOT 
set a continuation timer.
6. After that, watermark of ProcessFn itself progresses to infinity (because 
there's no hold anymore) and the pipeline terminates.

I suspect that in the Flink implementation, something is going wrong between 
steps 3 and 4. E.g. maybe the watermark hold isn't working (i.e. isn't 
preventing the watermark of ProcessFn from progressing to infinity); or maybe 
somehow the processing-time timer gets dropped for a different reason.

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)