[GitHub] beam pull request #1911: Fix broken licenses check for in jenkins files.

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/beam/pull/1911


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851096#comment-15851096
 ] 

ASF GitHub Bot commented on BEAM-886:
-

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1913

[BEAM-886] Rename NewDoFn to DoFn

R: @aaltay PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-866-rename-NewDoFn-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1913


commit dd30c503d27a1da0cafce605425b0d28b867e304
Author: Sourabh Bajaj 
Date:   2017-02-03T05:29:47Z

Rename NewDoFn to DoFn




> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1913: [BEAM-886] Rename NewDoFn to DoFn

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1913

[BEAM-886] Rename NewDoFn to DoFn

R: @aaltay PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-866-rename-NewDoFn-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1913


commit dd30c503d27a1da0cafce605425b0d28b867e304
Author: Sourabh Bajaj 
Date:   2017-02-03T05:29:47Z

Rename NewDoFn to DoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2519

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851064#comment-15851064
 ] 

Jean-Baptiste Onofré commented on BEAM-1384:


It seems it worked on the current build: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2519/org.apache.beam$beam-sdks-java-io-jms/

Waiting the end of the build to confirm.

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851054#comment-15851054
 ] 

Jean-Baptiste Onofré commented on BEAM-1384:


I'm waiting end of Jenkins build #2519. If it still happens, I will change the 
test to allow build to pass. It's weird that Jenkins worked on the PR.

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1384:
--

Assignee: Jean-Baptiste Onofré  (was: Davor Bonaci)

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1385) Apache RAT failure due to new website Jenkins job description DSL files

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-1385.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Apache RAT failure due to new website Jenkins job description DSL files
> ---
>
> Key: BEAM-1385
> URL: https://issues.apache.org/jira/browse/BEAM-1385
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1385) Apache RAT failure due to new website Jenkins job description DSL files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851041#comment-15851041
 ] 

ASF GitHub Bot commented on BEAM-1385:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1912


> Apache RAT failure due to new website Jenkins job description DSL files
> ---
>
> Key: BEAM-1385
> URL: https://issues.apache.org/jira/browse/BEAM-1385
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1912: [BEAM-1385] Add license header to website Jenkins j...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1912


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add license header to website Jenkins job descriptions

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 96a05a4b4 -> 08e58e1df


Add license header to website Jenkins job descriptions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/838ef04d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/838ef04d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/838ef04d

Branch: refs/heads/master
Commit: 838ef04da912091db6f83be2fe48a9a3525e8279
Parents: 96a05a4
Author: Kenneth Knowles 
Authored: Thu Feb 2 19:21:46 2017 -0800
Committer: Kenneth Knowles 
Committed: Thu Feb 2 19:21:46 2017 -0800

--
 .jenkins/job_beam_PreCommit_Website_Stage.groovy | 18 ++
 .jenkins/job_beam_PreCommit_Website_Test.groovy  | 18 ++
 2 files changed, 36 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/838ef04d/.jenkins/job_beam_PreCommit_Website_Stage.groovy
--
diff --git a/.jenkins/job_beam_PreCommit_Website_Stage.groovy 
b/.jenkins/job_beam_PreCommit_Website_Stage.groovy
index b63d40a..69be64d 100644
--- a/.jenkins/job_beam_PreCommit_Website_Stage.groovy
+++ b/.jenkins/job_beam_PreCommit_Website_Stage.groovy
@@ -1,3 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 import common_job_properties
 
 // Defines a job.

http://git-wip-us.apache.org/repos/asf/beam/blob/838ef04d/.jenkins/job_beam_PreCommit_Website_Test.groovy
--
diff --git a/.jenkins/job_beam_PreCommit_Website_Test.groovy 
b/.jenkins/job_beam_PreCommit_Website_Test.groovy
index ada089f..2b55374 100644
--- a/.jenkins/job_beam_PreCommit_Website_Test.groovy
+++ b/.jenkins/job_beam_PreCommit_Website_Test.groovy
@@ -1,3 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 import common_job_properties
 
 // Defines a job.

[2/2] beam git commit: This closes #1912: Add license header to website Jenkins job descriptions

[kenn]

This closes #1912: Add license header to website Jenkins job descriptions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/08e58e1d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/08e58e1d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/08e58e1d

Branch: refs/heads/master
Commit: 08e58e1dfd4cc21a24755d21b8241ca4488b0e83
Parents: 96a05a4 838ef04
Author: Kenneth Knowles 
Authored: Thu Feb 2 20:16:19 2017 -0800
Committer: Kenneth Knowles 
Committed: Thu Feb 2 20:16:19 2017 -0800

--
 .jenkins/job_beam_PreCommit_Website_Stage.groovy | 18 ++
 .jenkins/job_beam_PreCommit_Website_Test.groovy  | 18 ++
 2 files changed, 36 insertions(+)
--

[jira] [Commented] (BEAM-1385) Apache RAT failure due to new website Jenkins job description DSL files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851003#comment-15851003
 ] 

ASF GitHub Bot commented on BEAM-1385:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1912

[BEAM-1385] Add license header to website Jenkins job descriptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam groovy-RAT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1912


commit 838ef04da912091db6f83be2fe48a9a3525e8279
Author: Kenneth Knowles 
Date:   2017-02-03T03:21:46Z

Add license header to website Jenkins job descriptions




> Apache RAT failure due to new website Jenkins job description DSL files
> ---
>
> Key: BEAM-1385
> URL: https://issues.apache.org/jira/browse/BEAM-1385
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1385) Apache RAT failure due to new website Jenkins job description DSL files

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1385:
-

 Summary: Apache RAT failure due to new website Jenkins job 
description DSL files
 Key: BEAM-1385
 URL: https://issues.apache.org/jira/browse/BEAM-1385
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1912: [BEAM-1385] Add license header to website Jenkins j...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1912

[BEAM-1385] Add license header to website Jenkins job descriptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam groovy-RAT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1912


commit 838ef04da912091db6f83be2fe48a9a3525e8279
Author: Kenneth Knowles 
Date:   2017-02-03T03:21:46Z

Add license header to website Jenkins job descriptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1382) Test that standard coders encode the same across SDKs

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850968#comment-15850968
 ] 

Kenneth Knowles commented on BEAM-1382:
---

In addition to the description of the fact that {{Create}} ships encoded data 
up - which actually doesn't necessarily need to adhere to any standard coder - 
actually isolating coders to the Fn API would mean putting noop shims in the 
core graph construction bit of the SDK, which shouldn't depend on the Fn API 
model. If there's a gotcha where the core SDK has to depend on things that are 
part of the harness, then my thoughts don't apply, but as of now I don't know 
of any such issue.

> Test that standard coders encode the same across SDKs
> -
>
> Key: BEAM-1382
> URL: https://issues.apache.org/jira/browse/BEAM-1382
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api, beam-model-runner-api
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> There are a set of coders that cross the Fn Harness -- SDK Harness boundary. 
> These coders need to be consistent across SDKs (and the Fn harness as well). 
> We should have a standard, language-independent test spec and then actually 
> exercise it in both current SDKs (and all future ones).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1382) Test that standard coders encode the same across SDKs

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850966#comment-15850966
 ] 

Kenneth Knowles commented on BEAM-1382:
---

Noted on the PR that standardized coder formats actually come into play before 
the Fn API, though I understand the thought behind putting it in fn_api.

> Test that standard coders encode the same across SDKs
> -
>
> Key: BEAM-1382
> URL: https://issues.apache.org/jira/browse/BEAM-1382
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api, beam-model-runner-api
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> There are a set of coders that cross the Fn Harness -- SDK Harness boundary. 
> These coders need to be consistent across SDKs (and the Fn harness as well). 
> We should have a standard, language-independent test spec and then actually 
> exercise it in both current SDKs (and all future ones).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1384:
--
Summary: JmsIO postcommit failure: testAuthenticationWithBadPassword  (was: 
JmsIO postcommit failure: )

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Davor Bonaci
>Priority: Minor
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1384) JmsIO postcommit failure:

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1384:
-

 Summary: JmsIO postcommit failure: 
 Key: BEAM-1384
 URL: https://issues.apache.org/jira/browse/BEAM-1384
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: Davor Bonaci
Priority: Minor


Postcommit failure here: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/

It seems to be looking for a message which is actually there but buried deep in 
the cause stack.

Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1911: Fix broken licenses check for in jenkins files.

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1911

Fix broken licenses check for in jenkins files.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-licenses

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1911


commit 177c35fabb60497d0941aab2a6d2b5e982480b17
Author: Pei He 
Date:   2017-02-03T01:21:47Z

Fix broken licenses check for in jenkins files.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2517

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1383) Consistency in the Metrics examples

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1383:
---

 Summary: Consistency in the Metrics examples
 Key: BEAM-1383
 URL: https://issues.apache.org/jira/browse/BEAM-1383
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Pablo Estrada
Assignee: Ahmet Altay


snippets.py and wordcount.py initialize them in different places. Fix this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2518

[Apache Jenkins Server]

See 


Changes:

[davor] Move website jobs to main Beam repo to reduce duplication.

[altay] Add deprecation warning to OldDoFn

--
[...truncated 1615 lines...]
2017-02-03T01:13:21.854 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.pom
 (13 KB at 367.0 KB/sec)
2017-02-03T01:13:21.857 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/39/commons-parent-39.pom
2017-02-03T01:13:21.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/39/commons-parent-39.pom
 (61 KB at 1952.3 KB/sec)
2017-02-03T01:13:21.894 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.10/commons-compress-1.10.pom
2017-02-03T01:13:21.922 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.10/commons-compress-1.10.pom
 (13 KB at 438.1 KB/sec)
2017-02-03T01:13:21.925 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/38/commons-parent-38.pom
2017-02-03T01:13:21.959 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/38/commons-parent-38.pom
 (61 KB at 1786.0 KB/sec)
2017-02-03T01:13:21.966 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.6/doxia-decoration-model-1.6.pom
2017-02-03T01:13:21.996 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.6/doxia-decoration-model-1.6.pom
 (3 KB at 100.8 KB/sec)
2017-02-03T01:13:21.998 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sitetools/1.6/doxia-sitetools-1.6.pom
2017-02-03T01:13:22.028 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sitetools/1.6/doxia-sitetools-1.6.pom
 (18 KB at 582.3 KB/sec)
2017-02-03T01:13:22.032 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.pom
2017-02-03T01:13:22.059 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.pom
 (4 KB at 113.8 KB/sec)
2017-02-03T01:13:22.063 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.6/doxia-sink-api-1.6.pom
2017-02-03T01:13:22.091 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.6/doxia-sink-api-1.6.pom
 (2 KB at 55.4 KB/sec)
2017-02-03T01:13:22.093 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia/1.6/doxia-1.6.pom
2017-02-03T01:13:22.122 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia/1.6/doxia-1.6.pom
 (19 KB at 639.3 KB/sec)
2017-02-03T01:13:22.126 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.6/doxia-logging-api-1.6.pom
2017-02-03T01:13:22.154 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.6/doxia-logging-api-1.6.pom
 (2 KB at 55.6 KB/sec)
2017-02-03T01:13:22.157 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.6/doxia-site-renderer-1.6.pom
2017-02-03T01:13:22.185 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.6/doxia-site-renderer-1.6.pom
 (6 KB at 213.1 KB/sec)
2017-02-03T01:13:22.189 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.6/doxia-core-1.6.pom
2017-02-03T01:13:22.218 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.6/doxia-core-1.6.pom
 (5 KB at 141.5 KB/sec)
2017-02-03T01:13:22.222 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.6/doxia-module-xhtml-1.6.pom
2017-02-03T01:13:22.249 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.6/doxia-module-xhtml-1.6.pom
 (2 KB at 59.0 KB/sec)
2017-02-03T01:13:22.252 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-modules/1.6/doxia-modules-1.6.pom
2017-02-03T01:13:22.281 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-modules/1.6/doxia-modules-1.6.pom
 (3 KB at 88.3 KB/sec)
2017-02-03T01:13:22.285 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.6/doxia-module-fml-1.6.pom
2017-02-03T01:13:22.313 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.6/doxia-module-fml-1.6.pom
 (5 KB at 167.6 KB/sec)
2017-02-03T01:13:22.319 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/o

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850840#comment-15850840
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1902


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Add deprecation warning to OldDoFn

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 1533d7013 -> 96a05a4b4


Add deprecation warning to OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c0eef135
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c0eef135
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c0eef135

Branch: refs/heads/master
Commit: c0eef135e8fca960b7b16b37bbc3e13c52082a72
Parents: 1533d70
Author: Sourabh Bajaj 
Authored: Thu Feb 2 16:39:24 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 16:47:27 2017 -0800

--
 sdks/python/apache_beam/transforms/core.py | 11 ---
 sdks/python/apache_beam/typehints/typecheck.py |  8 +---
 2 files changed, 13 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c0eef135/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 9bafce5..bba6bd9 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -21,6 +21,7 @@ from __future__ import absolute_import
 
 import copy
 import inspect
+import warnings
 import types
 
 from apache_beam import pvalue
@@ -216,7 +217,7 @@ class NewDoFn(WithTypeHints, HasDisplayData):
 
 
 # TODO(Sourabh): Remove after migration to NewDoFn
-class DoFn(WithTypeHints, HasDisplayData):
+class OldDoFn(WithTypeHints, HasDisplayData):
   """A function object used by a transform with custom processing.
 
   The ParDo transform is such a transform. The ParDo.expand()
@@ -228,6 +229,10 @@ class DoFn(WithTypeHints, HasDisplayData):
   callable object using the CallableWrapperDoFn class.
   """
 
+  def __init__(self):
+warnings.warn('Use of OldDoFn is deprecated please use DoFn instead')
+super(OldDoFn, self).__init__()
+
   def default_label(self):
 return self.__class__.__name__
 
@@ -674,7 +679,7 @@ class ParDo(PTransformWithSideInputs):
   def __init__(self, fn_or_label, *args, **kwargs):
 super(ParDo, self).__init__(fn_or_label, *args, **kwargs)
 
-if not isinstance(self.fn, (DoFn, NewDoFn)):
+if not isinstance(self.fn, (OldDoFn, NewDoFn)):
   raise TypeError('ParDo must be called with a DoFn instance.')
 
   def default_type_hints(self):
@@ -685,7 +690,7 @@ class ParDo(PTransformWithSideInputs):
 self.fn.infer_output_type(input_type))
 
   def make_fn(self, fn):
-if isinstance(fn, (DoFn, NewDoFn)):
+if isinstance(fn, (OldDoFn, NewDoFn)):
   return fn
 return CallableWrapperDoFn(fn)
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c0eef135/sdks/python/apache_beam/typehints/typecheck.py
--
diff --git a/sdks/python/apache_beam/typehints/typecheck.py 
b/sdks/python/apache_beam/typehints/typecheck.py
index 7a10a5a..bc5583f 100644
--- a/sdks/python/apache_beam/typehints/typecheck.py
+++ b/sdks/python/apache_beam/typehints/typecheck.py
@@ -23,7 +23,7 @@ import sys
 import types
 
 from apache_beam.pvalue import SideOutputValue
-from apache_beam.transforms.core import DoFn
+from apache_beam.transforms.core import OldDoFn
 from apache_beam.transforms.core import NewDoFn
 from apache_beam.transforms.window import WindowedValue
 from apache_beam.typehints import check_constraint
@@ -35,7 +35,8 @@ from apache_beam.typehints.decorators import 
_check_instance_type
 from apache_beam.typehints.decorators import getcallargs_forhints
 
 
-class TypeCheckWrapperDoFn(DoFn):
+# TODO(Sourabh): Remove after migration to NewDoFn
+class TypeCheckWrapperDoFn(OldDoFn):
   """A wrapper around a DoFn which performs type-checking of input and output.
   """
 
@@ -123,7 +124,8 @@ class TypeCheckWrapperDoFn(DoFn):
   raise TypeCheckError, error_msg, sys.exc_info()[2]
 
 
-class OutputCheckWrapperDoFn(DoFn):
+# TODO(Sourabh): Remove after migration to NewDoFn
+class OutputCheckWrapperDoFn(OldDoFn):
   """A DoFn that verifies against common errors in the output type."""
 
   def __init__(self, dofn, full_label):

[2/2] beam git commit: This closes #1902

[altay]

This closes #1902


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/96a05a4b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/96a05a4b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/96a05a4b

Branch: refs/heads/master
Commit: 96a05a4b425f7107f282899ef93fd6dd44e22c8a
Parents: 1533d70 c0eef13
Author: Ahmet Altay 
Authored: Thu Feb 2 16:47:46 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 16:47:46 2017 -0800

--
 sdks/python/apache_beam/transforms/core.py | 11 ---
 sdks/python/apache_beam/typehints/typecheck.py |  8 +---
 2 files changed, 13 insertions(+), 6 deletions(-)
--

[GitHub] beam pull request #1902: [BEAM-886] Rename DoFn to OldDoFn and add deprecati...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1902


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1365) Bigquery dataset names should allow for . inside them

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850822#comment-15850822
 ] 

ASF GitHub Bot commented on BEAM-1365:
--

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1886


> Bigquery dataset names should allow for . inside them
> -
>
> Key: BEAM-1365
> URL: https://issues.apache.org/jira/browse/BEAM-1365
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> Bigquery datasets allow for . inside them but the regex in 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/bigquery.py#L305
>  doesn't allow for this. 
> Change the regex to 
> r'^((?P.+):)?(?P[\w\.]+)\.(?P[\w\$]+)$', table)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1886: [BEAM-1365] Fix the big query regex to account for ...

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1886


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1376) Move website job definitions to main Beam repo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850819#comment-15850819
 ] 

ASF GitHub Bot commented on BEAM-1376:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/143


> Move website job definitions to main Beam repo
> --
>
> Key: BEAM-1376
> URL: https://issues.apache.org/jira/browse/BEAM-1376
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #143: [BEAM-1376] Delete jobs due to move to main rep...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/143


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam-site git commit: Delete jobs due to move to main repo.

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 3ad83aefd -> d583b25d8


Delete jobs due to move to main repo.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b6ef4e4c
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b6ef4e4c
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b6ef4e4c

Branch: refs/heads/asf-site
Commit: b6ef4e4c9f73248c0ffc1d1732612bcd0da6662d
Parents: 3ad83ae
Author: Jason Kuster 
Authored: Wed Feb 1 16:45:33 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:41:13 2017 -0800

--
 .jenkins/common_job_properties.groovy | 106 -
 .jenkins/job_precommit_stage.groovy   |  62 -
 .jenkins/job_precommit_test.groovy|  47 -
 .jenkins/job_seed.groovy  |  32 -
 4 files changed, 247 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/b6ef4e4c/.jenkins/common_job_properties.groovy
--
diff --git a/.jenkins/common_job_properties.groovy 
b/.jenkins/common_job_properties.groovy
deleted file mode 100644
index 6bc3d14..000
--- a/.jenkins/common_job_properties.groovy
+++ /dev/null
@@ -1,106 +0,0 @@
-// Contains functions that help build Jenkins projects. Functions typically set
-// common properties that are shared among all Jenkins projects.
-class common_job_properties {
-
-  // Sets common top-level job properties.
-  static def setTopLevelJobProperties(def context) {
-
-// GitHub project.
-context.properties {
-  githubProjectUrl('https://github.com/apache/beam-site/')
-}
-
-// Restrict this project to run only on Jenkins executors dedicated to the
-// Apache Beam project.
-context.label('beam')
-
-// Discard old builds. Build records are only kept up to this number of 
days.
-context.logRotator {
-  daysToKeep(14)
-}
-
-// Source code management.
-context.scm {
-  git {
-remote {
-  url('https://github.com/apache/beam-site.git')
-  refspec('+refs/heads/*:refs/remotes/origin/* ' +
-  '+refs/pull/*:refs/remotes/origin/pr/*')
-}
-branch('${sha1}')
-extensions {
-  cleanAfterCheckout()
-}
-  }
-}
-
-context.parameters {
-  // This is a recommended setup if you want to run the job manually. The
-  // ${sha1} parameter needs to be provided, and defaults to the main 
branch.
-  stringParam(
-  'sha1',
-  'asf-site',
-  'Commit id or refname (eg: origin/pr/9/head) you want to build.')
-}
-
-context.wrappers {
-  // Abort the build if it's stuck for more minutes than specified.
-  timeout {
-absolute(30)
-abortBuild()
-  }
-}
-  }
-
-  // Sets the pull request build trigger.
-  static def setPullRequestBuildTrigger(def context,
-def commitStatusContext,
-def successComment = '--none--') {
-context.triggers {
-  githubPullRequest {
-admins(['asfbot'])
-useGitHubHooks()
-orgWhitelist(['apache'])
-allowMembersOfWhitelistedOrgsAsAdmin()
-permitAll()
-
-extensions {
-  commitStatus {
-// This is the name that will show up in the GitHub pull request UI
-// for this Jenkins project.
-delegate.context(commitStatusContext)
-  }
-
-  /*
-This section is disabled, because of jenkinsci/ghprb-plugin#417 
issue.
-For the time being, an equivalent configure section below is added.
-
-  // Comment messages after build completes.
-  buildStatus {
-completedStatus('SUCCESS', successComment)
-completedStatus('FAILURE', '--none--')
-completedStatus('ERROR', '--none--')
-  }
-  */
-}
-  }
-}
-
-// Comment messages after build completes.
-context.configure {
-  def messages = it / triggers / 
'org.jenkinsci.plugins.ghprb.GhprbTrigger' / extensions / 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildStatus' / messages
-  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
-message(successComment)
-result('SUCCESS')
-  }
-  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
-message('--none--')
-result('ERROR')
-  }
-  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
-message('--none--')
-result('FAILURE')
-  }
-}
-  }
-}

http://git-wip-us.apache.org/repo

[2/2] beam-site git commit: This closes #143

[davor]

This closes #143


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/d583b25d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/d583b25d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/d583b25d

Branch: refs/heads/asf-site
Commit: d583b25d85f7641e37e2bfe711e9ff6b1c0a2e35
Parents: 3ad83ae b6ef4e4
Author: Davor Bonaci 
Authored: Thu Feb 2 16:41:25 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:41:25 2017 -0800

--
 .jenkins/common_job_properties.groovy | 106 -
 .jenkins/job_precommit_stage.groovy   |  62 -
 .jenkins/job_precommit_test.groovy|  47 -
 .jenkins/job_seed.groovy  |  32 -
 4 files changed, 247 deletions(-)
--

[jira] [Commented] (BEAM-1376) Move website job definitions to main Beam repo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850816#comment-15850816
 ] 

ASF GitHub Bot commented on BEAM-1376:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1897


> Move website job definitions to main Beam repo
> --
>
> Key: BEAM-1376
> URL: https://issues.apache.org/jira/browse/BEAM-1376
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1382) Test that standard coders encode the same across SDKs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850814#comment-15850814
 ] 

ASF GitHub Bot commented on BEAM-1382:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1910

[BEAM-1382] Unify encoding of common coders across SDKs

Following Robert's original work in #1764,

* move the `standard_coders.yaml` file to the `common/fn_api` module in a 
test resource
* add the Java version of tests
* make the Python TupleCoder (and hence KVCoder) be more efficient in outer 
context and update spec so all 3 agree (spec, py, java).

Known issues:
- [ ] the python standard_coder_test ought to be failing to find the file, 
but `mvn clean install` locally does not fail. Still investigating.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam coder-crosssdk-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1910


commit 9b9af32927cef3fc40047f62dd27f1a8165b5cbc
Author: Dan Halperin 
Date:   2017-02-02T01:28:44Z

Implement tests that coders are equivalent across-sdks

Building off of the existing tests for Python SDK and unifying semantics.

commit 1cd5d58412a0c38967886920c244ad0fec766005
Author: Dan Halperin 
Date:   2017-02-02T23:27:54Z

fixup! lint fixes




> Test that standard coders encode the same across SDKs
> -
>
> Key: BEAM-1382
> URL: https://issues.apache.org/jira/browse/BEAM-1382
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api, beam-model-runner-api
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> There are a set of coders that cross the Fn Harness -- SDK Harness boundary. 
> These coders need to be consistent across SDKs (and the Fn harness as well). 
> We should have a standard, language-independent test spec and then actually 
> exercise it in both current SDKs (and all future ones).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1897: [BEAM-1376] Move website jobs to main Beam repo to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1897


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1910: [BEAM-1382] Unify encoding of common coders across ...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1910

[BEAM-1382] Unify encoding of common coders across SDKs

Following Robert's original work in #1764,

* move the `standard_coders.yaml` file to the `common/fn_api` module in a 
test resource
* add the Java version of tests
* make the Python TupleCoder (and hence KVCoder) be more efficient in outer 
context and update spec so all 3 agree (spec, py, java).

Known issues:
- [ ] the python standard_coder_test ought to be failing to find the file, 
but `mvn clean install` locally does not fail. Still investigating.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam coder-crosssdk-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1910


commit 9b9af32927cef3fc40047f62dd27f1a8165b5cbc
Author: Dan Halperin 
Date:   2017-02-02T01:28:44Z

Implement tests that coders are equivalent across-sdks

Building off of the existing tests for Python SDK and unifying semantics.

commit 1cd5d58412a0c38967886920c244ad0fec766005
Author: Dan Halperin 
Date:   2017-02-02T23:27:54Z

fixup! lint fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1897

[davor]

This closes #1897


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1533d701
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1533d701
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1533d701

Branch: refs/heads/master
Commit: 1533d70137a77db497ba6dcebcb421c1988ec1df
Parents: c0910fd 5dc83f5
Author: Davor Bonaci 
Authored: Thu Feb 2 16:39:00 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:39:00 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 40 +
 ...job_beam_PostCommit_Java_MavenInstall.groovy |  2 +-
 ...ostCommit_Java_RunnableOnService_Apex.groovy |  2 +-
 ...ommit_Java_RunnableOnService_Dataflow.groovy |  2 +-
 ...stCommit_Java_RunnableOnService_Flink.groovy |  2 +-
 ...ommit_Java_RunnableOnService_Gearpump.groovy |  4 +-
 ...stCommit_Java_RunnableOnService_Spark.groovy |  2 +-
 .../job_beam_PostCommit_Python_Verify.groovy|  2 +-
 .../job_beam_PreCommit_Java_MavenInstall.groovy |  2 +-
 .../job_beam_PreCommit_Website_Stage.groovy | 62 
 .jenkins/job_beam_PreCommit_Website_Test.groovy | 47 +++
 .../job_beam_Release_NightlySnapshot.groovy |  2 +-
 .jenkins/job_seed.groovy| 11 ++--
 13 files changed, 154 insertions(+), 26 deletions(-)
--

[1/2] beam git commit: Move website jobs to main Beam repo to reduce duplication.

[davor]

Repository: beam
Updated Branches:
  refs/heads/master c0910fdf4 -> 1533d7013


Move website jobs to main Beam repo to reduce duplication.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5dc83f57
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5dc83f57
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5dc83f57

Branch: refs/heads/master
Commit: 5dc83f577e7b4275475556bf3bd0212bf69e3d35
Parents: c0910fd
Author: Jason Kuster 
Authored: Wed Feb 1 16:39:40 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:38:47 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 40 +
 ...job_beam_PostCommit_Java_MavenInstall.groovy |  2 +-
 ...ostCommit_Java_RunnableOnService_Apex.groovy |  2 +-
 ...ommit_Java_RunnableOnService_Dataflow.groovy |  2 +-
 ...stCommit_Java_RunnableOnService_Flink.groovy |  2 +-
 ...ommit_Java_RunnableOnService_Gearpump.groovy |  4 +-
 ...stCommit_Java_RunnableOnService_Spark.groovy |  2 +-
 .../job_beam_PostCommit_Python_Verify.groovy|  2 +-
 .../job_beam_PreCommit_Java_MavenInstall.groovy |  2 +-
 .../job_beam_PreCommit_Website_Stage.groovy | 62 
 .jenkins/job_beam_PreCommit_Website_Test.groovy | 47 +++
 .../job_beam_Release_NightlySnapshot.groovy |  2 +-
 .jenkins/job_seed.groovy| 11 ++--
 13 files changed, 154 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5dc83f57/.jenkins/common_job_properties.groovy
--
diff --git a/.jenkins/common_job_properties.groovy 
b/.jenkins/common_job_properties.groovy
index f1f2b7f..eee1cda 100644
--- a/.jenkins/common_job_properties.groovy
+++ b/.jenkins/common_job_properties.groovy
@@ -20,14 +20,27 @@
 // common properties that are shared among all Jenkins projects.
 class common_job_properties {
 
-  // Sets common top-level job properties.
-  static def setTopLevelJobProperties(def context,
-  def default_branch = 'master',
-  def default_timeout = 100) {
+  // Sets common top-level job properties for website repository jobs.
+  static def setTopLevelWebsiteJobProperties(def context) {
+setTopLevelJobProperties(context, 'beam-site', 'asf-site', 30)
+  }
+
+  // Sets common top-level job properties for main repository jobs.
+  static def setTopLevelMainJobProperties(def context,
+  def default_branch = 'master') {
+setTopLevelJobProperties(context, 'beam', default_branch, 100)
+  }
+
+  // Sets common top-level job properties. Should be accessed through one of 
the
+  // above methods to protect jobs from internal details of param defaults.
+  private static def setTopLevelJobProperties(def context,
+  def repository_name,
+  def default_branch,
+  def default_timeout) {
 
 // GitHub project.
 context.properties {
-  githubProjectUrl('https://github.com/apache/beam/')
+  githubProjectUrl('https://github.com/apache/' + repository_name + '/')
 }
 
 // Set JDK version.
@@ -46,7 +59,7 @@ class common_job_properties {
 context.scm {
   git {
 remote {
-  url('https://github.com/apache/beam.git')
+  url('https://github.com/apache/' + repository_name + '.git')
   refspec('+refs/heads/*:refs/remotes/origin/* ' +
   '+refs/pull/*:refs/remotes/origin/pr/*')
 }
@@ -83,10 +96,11 @@ class common_job_properties {
 }
   }
 
-  // Sets the pull request build trigger.
-  static def setPullRequestBuildTrigger(def context,
-def commitStatusContext,
-def successComment = '--none--') {
+  // Sets the pull request build trigger. Accessed through precommit methods
+  // below to insulate callers from internal parameter defaults.
+  private static def setPullRequestBuildTrigger(def context,
+def commitStatusContext,
+def successComment = 
'--none--') {
 context.triggers {
   githubPullRequest {
 admins(['asfbot'])
@@ -150,9 +164,11 @@ class common_job_properties {
   }
 
   // Sets common config for PreCommit jobs.
-  static def setPreCommit(def context, comment) {
+  static def setPreCommit(def context,
+  def commitStatusName,
+  def successComment = '--none--') {
 // Set pull request build trigger.
-setPullRequestBuildTrigger(context, comment)
+setPul

[jira] [Closed] (BEAM-1370) AfterWatermarkEarlyAndLate does not invoke the onMerge of the early trigger

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh closed BEAM-1370.
-
   Resolution: Fixed
Fix Version/s: 0.5.0

fixed in {{0a901d9489afec98c4b0d6db02e4837030e2a404}}

> AfterWatermarkEarlyAndLate does not invoke the onMerge of the early trigger
> ---
>
> Key: BEAM-1370
> URL: https://issues.apache.org/jira/browse/BEAM-1370
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.5.0
>
>
> When a window merges in AfterWatermarkStateMachine, if the early trigger is 
> not finished the state machine should invoke its onMerge. This ensures, for 
> example, that with merging windows with a processing time early trigger, that 
> trigger is propagated to the merged windows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1382) Test that standard coders encode the same across SDKs

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1382:
-

 Summary: Test that standard coders encode the same across SDKs
 Key: BEAM-1382
 URL: https://issues.apache.org/jira/browse/BEAM-1382
 Project: Beam
  Issue Type: Bug
  Components: beam-model-fn-api, beam-model-runner-api
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin
 Fix For: Not applicable


There are a set of coders that cross the Fn Harness -- SDK Harness boundary. 
These coders need to be consistent across SDKs (and the Fn harness as well). We 
should have a standard, language-independent test spec and then actually 
exercise it in both current SDKs (and all future ones).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850792#comment-15850792
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1896


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Migrate from OldDoFn to NewDoFn

[altay]

Repository: beam
Updated Branches:
  refs/heads/master a88f59063 -> c0910fdf4


Migrate from OldDoFn to NewDoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a08eff5e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a08eff5e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a08eff5e

Branch: refs/heads/master
Commit: a08eff5e7c29f00d9d6f00916a39a084e1602991
Parents: a88f590
Author: Sourabh Bajaj 
Authored: Thu Feb 2 15:45:00 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 16:06:44 2017 -0800

--
 .../examples/complete/top_wikipedia_sessions.py | 32 -
 .../examples/cookbook/datastore_wordcount.py|  8 +--
 .../examples/cookbook/multiple_output_pardo.py  | 10 +--
 .../apache_beam/examples/snippets/snippets.py   | 32 -
 .../examples/snippets/snippets_test.py  | 58 +++
 sdks/python/apache_beam/examples/wordcount.py   |  8 +--
 .../apache_beam/examples/wordcount_debugging.py |  8 +--
 .../apache_beam/io/datastore/v1/datastoreio.py  | 26 ---
 .../io/datastore/v1/datastoreio_test.py | 32 +++--
 sdks/python/apache_beam/io/iobase.py| 23 +++---
 .../consumer_tracking_pipeline_visitor_test.py  | 18 ++---
 .../runners/direct/helper_transforms.py | 20 +++---
 sdks/python/apache_beam/runners/runner_test.py  | 16 ++---
 sdks/python/apache_beam/transforms/core.py  | 61 
 .../apache_beam/transforms/display_test.py  | 12 ++--
 .../apache_beam/transforms/ptransform_test.py   | 75 ++--
 .../apache_beam/transforms/window_test.py   | 41 ++-
 .../typehints/typed_pipeline_test.py| 12 ++--
 18 files changed, 240 insertions(+), 252 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a08eff5e/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py 
b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
index d19f66d..43a4ee2 100644
--- a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
+++ b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
@@ -45,9 +45,11 @@ import logging
 
 import apache_beam as beam
 from apache_beam import combiners
-from apache_beam import window
 from apache_beam.io import ReadFromText
 from apache_beam.io import WriteToText
+from apache_beam.transforms.window import FixedWindows
+from apache_beam.transforms.window import Sessions
+from apache_beam.transforms.window import TimestampedValue
 from apache_beam.utils.pipeline_options import PipelineOptions
 from apache_beam.utils.pipeline_options import SetupOptions
 
@@ -57,15 +59,15 @@ THIRTY_DAYS_IN_SECONDS = 30 * 24 * ONE_HOUR_IN_SECONDS
 MAX_TIMESTAMP = 0x7fff
 
 
-class ExtractUserAndTimestampDoFn(beam.DoFn):
+class ExtractUserAndTimestampDoFn(beam.NewDoFn):
   """Extracts user and timestamp representing a Wikipedia edit."""
 
-  def process(self, context):
-table_row = json.loads(context.element)
+  def process(self, element):
+table_row = json.loads(element)
 if 'contributor_username' in table_row:
   user_name = table_row['contributor_username']
   timestamp = table_row['timestamp']
-  yield window.TimestampedValue(user_name, timestamp)
+  yield TimestampedValue(user_name, timestamp)
 
 
 class ComputeSessions(beam.PTransform):
@@ -81,7 +83,7 @@ class ComputeSessions(beam.PTransform):
   def expand(self, pcoll):
 return (pcoll
 | 'ComputeSessionsWindow' >> beam.WindowInto(
-window.Sessions(gap_size=ONE_HOUR_IN_SECONDS))
+Sessions(gap_size=ONE_HOUR_IN_SECONDS))
 | combiners.Count.PerElement())
 
 
@@ -94,30 +96,28 @@ class TopPerMonth(beam.PTransform):
   def expand(self, pcoll):
 return (pcoll
 | 'TopPerMonthWindow' >> beam.WindowInto(
-window.FixedWindows(size=THIRTY_DAYS_IN_SECONDS))
+FixedWindows(size=THIRTY_DAYS_IN_SECONDS))
 | 'Top' >> combiners.core.CombineGlobally(
 combiners.TopCombineFn(
 10, lambda first, second: first[1] < second[1]))
 .without_defaults())
 
 
-class SessionsToStringsDoFn(beam.DoFn):
+class SessionsToStringsDoFn(beam.NewDoFn):
   """Adds the session information to be part of the key."""
 
-  def process(self, context):
-yield (context.element[0] + ' : ' +
-   ', '.join([str(w) for w in context.windows]), context.element[1])
+  def process(self, element, window=beam.NewDoFn.WindowParam):
+yield (element[0] + ' : ' + str(window), element[1])
 
 
-class FormatOutputDoFn(beam.DoFn):
+class FormatOutputDoFn(beam.NewDoFn):
  

[GitHub] beam pull request #1896: [BEAM-886] Migrate from OldDoFn to NewDoFn

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1896


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1896

[altay]

This closes #1896


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c0910fdf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c0910fdf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c0910fdf

Branch: refs/heads/master
Commit: c0910fdf4939a4db52b4e7c49e3593b6da31694a
Parents: a88f590 a08eff5
Author: Ahmet Altay 
Authored: Thu Feb 2 16:06:55 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 16:06:55 2017 -0800

--
 .../examples/complete/top_wikipedia_sessions.py | 32 -
 .../examples/cookbook/datastore_wordcount.py|  8 +--
 .../examples/cookbook/multiple_output_pardo.py  | 10 +--
 .../apache_beam/examples/snippets/snippets.py   | 32 -
 .../examples/snippets/snippets_test.py  | 58 +++
 sdks/python/apache_beam/examples/wordcount.py   |  8 +--
 .../apache_beam/examples/wordcount_debugging.py |  8 +--
 .../apache_beam/io/datastore/v1/datastoreio.py  | 26 ---
 .../io/datastore/v1/datastoreio_test.py | 32 +++--
 sdks/python/apache_beam/io/iobase.py| 23 +++---
 .../consumer_tracking_pipeline_visitor_test.py  | 18 ++---
 .../runners/direct/helper_transforms.py | 20 +++---
 sdks/python/apache_beam/runners/runner_test.py  | 16 ++---
 sdks/python/apache_beam/transforms/core.py  | 61 
 .../apache_beam/transforms/display_test.py  | 12 ++--
 .../apache_beam/transforms/ptransform_test.py   | 75 ++--
 .../apache_beam/transforms/window_test.py   | 41 ++-
 .../typehints/typed_pipeline_test.py| 12 ++--
 18 files changed, 240 insertions(+), 252 deletions(-)
--

[jira] [Resolved] (BEAM-1380) Typo in programming guide

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1380.

   Resolution: Fixed
Fix Version/s: Not applicable

> Typo in programming guide
> -
>
> Key: BEAM-1380
> URL: https://issues.apache.org/jira/browse/BEAM-1380
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Bill Neubauer
>Assignee: Davor Bonaci
>Priority: Trivial
>  Labels: easyfix
> Fix For: Not applicable
>
>
> https://beam.apache.org/documentation/programming-guide/
> s/perfroms/performs/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1380) Typo in programming guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850777#comment-15850777
 ] 

ASF GitHub Bot commented on BEAM-1380:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/145


> Typo in programming guide
> -
>
> Key: BEAM-1380
> URL: https://issues.apache.org/jira/browse/BEAM-1380
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Bill Neubauer
>Assignee: Davor Bonaci
>Priority: Trivial
>  Labels: easyfix
>
> https://beam.apache.org/documentation/programming-guide/
> s/perfroms/performs/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/debe8c2c
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/debe8c2c
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/debe8c2c

Branch: refs/heads/asf-site
Commit: debe8c2cedc871d4c1c5a1b3ba3a91c94436a1fb
Parents: da5ee69
Author: Davor Bonaci 
Authored: Thu Feb 2 16:06:38 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:06:38 2017 -0800

--
 content/documentation/programming-guide/index.html |  2 +-
 content/get-started/wordcount-example/index.html   | 17 -
 2 files changed, 9 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/debe8c2c/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index d6ccda9..b301096 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -208,7 +208,7 @@
 PCollection: A PCollection represents a distributed data set 
that your Beam pipeline operates on. The data set can be bounded, 
meaning it comes from a fixed source like a file, or unbounded, 
meaning it comes from a continuously updating source via a subscription or 
other mechanism. Your pipeline typically creates an initial PCollection by reading data from an external 
data source, but you can also create a PCollection from in-memory data within your 
driver program. From there, PCollections 
are the inputs and outputs for each step in your pipeline.
   
   
-Transform: A Transform represents a data processing 
operation, or a step, in your pipeline. Every Transform takes one or more PCollection objects as input, perfroms a 
processing function that you provide on the elements of that PCollection, and produces one or more output 
PCollection objects.
+Transform: A Transform represents a data processing 
operation, or a step, in your pipeline. Every Transform takes one or more PCollection objects as input, performs a 
processing function that you provide on the elements of that PCollection, and produces one or more output 
PCollection objects.
   
   
 I/O Source and Sink: Beam provides Source and Sink APIs to represent reading and writing 
data, respectively. Source encapsulates 
the code necessary to read data into your Beam pipeline from some external 
source, such as cloud file storage or a subscription to a streaming data 
source. Sink likewise encapsulates the 
code necessary to write the elements of a PCollection to an external data sink.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/debe8c2c/content/get-started/wordcount-example/index.html
--
diff --git a/content/get-started/wordcount-example/index.html 
b/content/get-started/wordcount-example/index.html
index 1cc4f9a..eb4deec 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -556,13 +556,12 @@ Figure 1: The pipeline data flow.
 class FilterTextFn(beam.DoFn):
   """A DoFn that filters for a specific key based on a regular 
expression."""
 
-  # A custom aggregator can track values in your pipeline as 
it runs. Create
-  # custom aggregators matched_word and unmatched_words.
-  matched_words = beam.Aggregator('matched_words')
-  umatched_words = beam.Aggregator('umatched_words')
-
   def __init__(self, pattern):
 self.pattern = pattern
+# A custom metric can track values in your pipeline as it 
runs. Create
+# custom metrics matched_word and unmatched_words.
+self.matched_words = Metrics.counter(self.__class__, 'matched_words')
+self.umatched_words = Metrics.counter(self.__class__, 'umatched_words')
 
   def process(self, context):
 word, _ = context.element
@@ -572,8 +571,8 @@ Figure 1: The pipeline data flow.
   # Logging UI.
   logging.info('Matched 
%s', word)
 
-  # Add 1 to the custom aggregator matched_words
-  context.aggregate_to(self.matched_words, 1)
+  # Add 1 to the custom metric counter matched_words
+  self.matched_words.inc()
   yield context.element
 else:
   # Log at the "DEBUG" level each element that is not 
matched. Different
@@ -583,8 +582,8 @@ Figure 1: The pipeline data flow.
   # Logger. This log message will not be visible in the 
Cloud Logger.
   logging.debug('Did not match 
%s', word)
 
-  # Add 1 to the custom aggregator umatched_words
-  context.aggregate_to(self.umatched_words, 1)
+  # Add 1 to the custom metric counter 
umatched_words
+  self.umatched_words.inc()
 
 
 

[1/3] beam-site git commit: Fix typo in programming-guide.md

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site a53de48ce -> 3ad83aefd


Fix typo in programming-guide.md

Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/da5ee690
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/da5ee690
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/da5ee690

Branch: refs/heads/asf-site
Commit: da5ee6907b1d54d32cf66b85dbd23ec9ae668474
Parents: a53de48
Author: Davor Bonaci 
Authored: Thu Feb 2 15:06:29 2017 -0800
Committer: GitHub 
Committed: Thu Feb 2 15:06:29 2017 -0800

--
 src/documentation/programming-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/da5ee690/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 9846929..3cd64f0 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -55,7 +55,7 @@ The Beam SDKs provide a number of abstractions that simplify 
the mechanics of la
 
 * `PCollection`: A `PCollection` represents a distributed data set that your 
Beam pipeline operates on. The data set can be *bounded*, meaning it comes from 
a fixed source like a file, or *unbounded*, meaning it comes from a 
continuously updating source via a subscription or other mechanism. Your 
pipeline typically creates an initial `PCollection` by reading data from an 
external data source, but you can also create a `PCollection` from in-memory 
data within your driver program. From there, `PCollection`s are the inputs and 
outputs for each step in your pipeline.
 
-* `Transform`: A `Transform` represents a data processing operation, or a 
step, in your pipeline. Every `Transform` takes one or more `PCollection` 
objects as input, perfroms a processing function that you provide on the 
elements of that `PCollection`, and produces one or more output `PCollection` 
objects. 
+* `Transform`: A `Transform` represents a data processing operation, or a 
step, in your pipeline. Every `Transform` takes one or more `PCollection` 
objects as input, performs a processing function that you provide on the 
elements of that `PCollection`, and produces one or more output `PCollection` 
objects. 
 
 * I/O `Source` and `Sink`: Beam provides `Source` and `Sink` APIs to represent 
reading and writing data, respectively. `Source` encapsulates the code 
necessary to read data into your Beam pipeline from some external source, such 
as cloud file storage or a subscription to a streaming data source. `Sink` 
likewise encapsulates the code necessary to write the elements of a 
`PCollection` to an external data sink.
 

[GitHub] beam-site pull request #145: [BEAM-1380] Fix typo in programming-guide.md

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/145


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #145

[davor]

This closes #145


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/3ad83aef
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/3ad83aef
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/3ad83aef

Branch: refs/heads/asf-site
Commit: 3ad83aefd4a2064e2c793fc4707ffe3d7d5db6d0
Parents: a53de48 debe8c2
Author: Davor Bonaci 
Authored: Thu Feb 2 16:06:38 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 16:06:38 2017 -0800

--
 content/documentation/programming-guide/index.html |  2 +-
 content/get-started/wordcount-example/index.html   | 17 -
 src/documentation/programming-guide.md |  2 +-
 3 files changed, 10 insertions(+), 11 deletions(-)
--

[GitHub] beam pull request #1894: [BEAM-886] Migrate CallableWrapperDoFn to use the N...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1894


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850769#comment-15850769
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1894


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Migrate CallableWrapperDoFn to use the NewDoFn type

[altay]

Repository: beam
Updated Branches:
  refs/heads/master c01ed083e -> a88f59063


Migrate CallableWrapperDoFn to use the NewDoFn type


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/926f8687
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/926f8687
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/926f8687

Branch: refs/heads/master
Commit: 926f86877346a68b95b1c618b69553d1c53e1c98
Parents: c01ed08
Author: Sourabh Bajaj 
Authored: Thu Feb 2 14:48:57 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 15:51:26 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py   | 16 
 sdks/python/apache_beam/transforms/core.py | 11 ++-
 2 files changed, 22 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/926f8687/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index 833293f..95b55b9 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -110,6 +110,22 @@ class PipelineTest(unittest.TestCase):
 assert_that(pcoll3, equal_to([14, 15, 16]), label='pcoll3')
 pipeline.run()
 
+  def test_flatmap_builtin(self):
+pipeline = TestPipeline()
+pcoll = pipeline | 'label1' >> Create([1, 2, 3])
+assert_that(pcoll, equal_to([1, 2, 3]))
+
+pcoll2 = pcoll | 'do' >> FlatMap(lambda x: [x + 10])
+assert_that(pcoll2, equal_to([11, 12, 13]), label='pcoll2')
+
+pcoll3 = pcoll2 | 'm1' >> Map(lambda x: [x, 12])
+assert_that(pcoll3,
+equal_to([[11, 12], [12, 12], [13, 12]]), label='pcoll3')
+
+pcoll4 = pcoll3 | 'do2' >> FlatMap(set)
+assert_that(pcoll4, equal_to([11, 12, 12, 12, 13]), label='pcoll4')
+pipeline.run()
+
   def test_create_singleton_pcollection(self):
 pipeline = TestPipeline()
 pcoll = pipeline | 'label' >> Create([[1, 2, 3]])

http://git-wip-us.apache.org/repos/asf/beam/blob/926f8687/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 20126d3..f69511f 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -304,7 +304,7 @@ def _fn_takes_side_inputs(fn):
   return len(argspec.args) > 1 + is_bound or argspec.varargs or 
argspec.keywords
 
 
-class CallableWrapperDoFn(DoFn):
+class CallableWrapperDoFn(NewDoFn):
   """A DoFn (function) object wrapping a callable object.
 
   The purpose of this class is to conveniently wrap simple functions and use
@@ -324,11 +324,12 @@ class CallableWrapperDoFn(DoFn):
   raise TypeError('Expected a callable object instead of: %r' % fn)
 
 self._fn = fn
-if _fn_takes_side_inputs(fn):
-  self.process = lambda context, *args, **kwargs: fn(
-  context.element, *args, **kwargs)
+if isinstance(fn, (
+types.BuiltinFunctionType, types.MethodType, types.FunctionType)):
+  self.process = fn
 else:
-  self.process = lambda context: fn(context.element)
+  # For cases such as set / list where fn is callable but not a function
+  self.process = lambda element: fn(element)
 
 super(CallableWrapperDoFn, self).__init__()
 

[2/2] beam git commit: This closes #1894

[altay]

This closes #1894


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a88f5906
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a88f5906
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a88f5906

Branch: refs/heads/master
Commit: a88f590638425e78960b81227eba2fea0cf5876c
Parents: c01ed08 926f868
Author: Ahmet Altay 
Authored: Thu Feb 2 15:51:43 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 15:51:43 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py   | 16 
 sdks/python/apache_beam/transforms/core.py | 11 ++-
 2 files changed, 22 insertions(+), 5 deletions(-)
--

[jira] [Commented] (BEAM-1381) Implement DataflowMetrics.query method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850721#comment-15850721
 ] 

ASF GitHub Bot commented on BEAM-1381:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1909


> Implement DataflowMetrics.query method
> --
>
> Key: BEAM-1381
> URL: https://issues.apache.org/jira/browse/BEAM-1381
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Ahmet Altay
>
> Once metrics can bequeried from the dataflow service, we must implement the 
> logic for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1909: [BEAM-1381] Adding metrics() to DataflowPipelineRes...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1909


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1909

[altay]

This closes #1909


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c01ed083
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c01ed083
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c01ed083

Branch: refs/heads/master
Commit: c01ed083e0a412500104a5c9a63c508ae1dea7b8
Parents: 1a6f2e8 ac4185e
Author: Ahmet Altay 
Authored: Thu Feb 2 15:08:28 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 15:08:28 2017 -0800

--
 .../runners/dataflow/dataflow_metrics.py| 33 
 .../runners/dataflow/dataflow_metrics_test.py   | 20 
 .../apache_beam/runners/dataflow_runner.py  |  4 +++
 .../apache_beam/runners/dataflow_runner_test.py |  5 +++
 sdks/python/apache_beam/runners/runner.py   |  8 +
 5 files changed, 70 insertions(+)
--

[1/2] beam git commit: Adding metrics() to DataflowPipelineResult.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 1a6f2e8f6 -> c01ed083e


Adding metrics() to DataflowPipelineResult.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ac4185ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ac4185ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ac4185ed

Branch: refs/heads/master
Commit: ac4185ed3952397590947d79832fa0b1507cce19
Parents: 1a6f2e8
Author: Pablo 
Authored: Thu Feb 2 13:42:58 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 15:08:22 2017 -0800

--
 .../runners/dataflow/dataflow_metrics.py| 33 
 .../runners/dataflow/dataflow_metrics_test.py   | 20 
 .../apache_beam/runners/dataflow_runner.py  |  4 +++
 .../apache_beam/runners/dataflow_runner_test.py |  5 +++
 sdks/python/apache_beam/runners/runner.py   |  8 +
 5 files changed, 70 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ac4185ed/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
new file mode 100644
index 000..1d86f2f
--- /dev/null
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
@@ -0,0 +1,33 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+DataflowRunner implementation of MetricResults. It is in charge of
+responding to queries of current metrics by going to the dataflow
+service.
+"""
+
+from apache_beam.metrics.metric import MetricResults
+
+
+# TODO(pabloem)(JIRA-1381) Implement this once metrics are queriable from
+# dataflow service
+class DataflowMetrics(MetricResults):
+
+  def query(self, filter=None):
+return {'counters': [],
+'distributions': []}

http://git-wip-us.apache.org/repos/asf/beam/blob/ac4185ed/sdks/python/apache_beam/runners/dataflow/dataflow_metrics_test.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics_test.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics_test.py
new file mode 100644
index 000..5475ac7
--- /dev/null
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics_test.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Tests corresponding to the DataflowRunner implementation of MetricsResult,
+the DataflowMetrics class.
+"""

http://git-wip-us.apache.org/repos/asf/beam/blob/ac4185ed/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index abcc764..f02e24b 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -32,6 +32,7 @@ from apache_beam import pvalue
 from apache_beam.internal import json_value
 from apache_beam.internal import pickler
 from apache_beam.pvalue import PCollectionView
+from apache_beam.runners.dataflow.dataflow_metrics import DataflowMetrics
 from apache_beam.runners.runner import PipelineResult
 from apache_beam.runners.r

[jira] [Commented] (BEAM-1380) Typo in programming guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850702#comment-15850702
 ] 

ASF GitHub Bot commented on BEAM-1380:
--

GitHub user davorbonaci opened a pull request:

https://github.com/apache/beam-site/pull/145

[BEAM-1380] Fix typo in programming-guide.md

No review needed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #145


commit da5ee6907b1d54d32cf66b85dbd23ec9ae668474
Author: Davor Bonaci 
Date:   2017-02-02T23:06:29Z

Fix typo in programming-guide.md




> Typo in programming guide
> -
>
> Key: BEAM-1380
> URL: https://issues.apache.org/jira/browse/BEAM-1380
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Bill Neubauer
>Assignee: Davor Bonaci
>Priority: Trivial
>  Labels: easyfix
>
> https://beam.apache.org/documentation/programming-guide/
> s/perfroms/performs/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #145: [BEAM-1380] Fix typo in programming-guide.md

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/beam-site/pull/145

[BEAM-1380] Fix typo in programming-guide.md

No review needed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #145


commit da5ee6907b1d54d32cf66b85dbd23ec9ae668474
Author: Davor Bonaci 
Date:   2017-02-02T23:06:29Z

Fix typo in programming-guide.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1381) Implement DataflowMetrics.query method

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1381:
---

 Summary: Implement DataflowMetrics.query method
 Key: BEAM-1381
 URL: https://issues.apache.org/jira/browse/BEAM-1381
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Pablo Estrada
Assignee: Ahmet Altay


Once metrics can bequeried from the dataflow service, we must implement the 
logic for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850568#comment-15850568
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1869


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1869: [BEAM-886] Some performance improvements to NewDoFn

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1869


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1869

[altay]

This closes #1869


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1a6f2e8f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1a6f2e8f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1a6f2e8f

Branch: refs/heads/master
Commit: 1a6f2e8f62eddcc3351458eac6e6d9a97f0ec094
Parents: 70b16c7 956b81c
Author: Ahmet Altay 
Authored: Thu Feb 2 13:37:05 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 13:37:05 2017 -0800

--
 sdks/python/apache_beam/runners/common.pxd |   9 +-
 sdks/python/apache_beam/runners/common.py  | 157 +++-
 2 files changed, 107 insertions(+), 59 deletions(-)
--

[1/2] beam git commit: Some performance improvements to NewDoFn

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 70b16c74d -> 1a6f2e8f6


Some performance improvements to NewDoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/956b81cf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/956b81cf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/956b81cf

Branch: refs/heads/master
Commit: 956b81cfa18281366c2f6bc41b02b099ec37d210
Parents: 70b16c7
Author: Sourabh Bajaj 
Authored: Mon Jan 30 11:16:16 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 13:36:58 2017 -0800

--
 sdks/python/apache_beam/runners/common.pxd |   9 +-
 sdks/python/apache_beam/runners/common.py  | 157 +++-
 2 files changed, 107 insertions(+), 59 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/956b81cf/sdks/python/apache_beam/runners/common.pxd
--
diff --git a/sdks/python/apache_beam/runners/common.pxd 
b/sdks/python/apache_beam/runners/common.pxd
index f41b313..dbb08f0 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -38,17 +38,22 @@ cdef class DoFnRunner(Receiver):
   cdef LoggingContext logging_context
   cdef object step_name
   cdef bint is_new_dofn
-  cdef object args
+  cdef list args
   cdef dict kwargs
-  cdef object side_inputs
   cdef ScopedMetricsContainer scoped_metrics_container
+  cdef list side_inputs
   cdef bint has_windowed_side_inputs
+  cdef list placeholders
+  cdef bint simple_process
 
   cdef Receiver main_receivers
 
   cpdef process(self, WindowedValue element)
   cdef old_dofn_process(self, WindowedValue element)
   cdef new_dofn_process(self, WindowedValue element)
+  cdef new_dofn_simple_process(self, WindowedValue element)
+  cdef _new_dofn_window_process(
+  self, WindowedValue element, list args, dict kwargs, object window)
 
   @cython.locals(windowed_value=WindowedValue)
   cpdef _process_outputs(self, WindowedValue element, results)

http://git-wip-us.apache.org/repos/asf/beam/blob/956b81cf/sdks/python/apache_beam/runners/common.py
--
diff --git a/sdks/python/apache_beam/runners/common.py 
b/sdks/python/apache_beam/runners/common.py
index dbbd9ba..0089f34 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -25,9 +25,9 @@ from apache_beam.internal import util
 from apache_beam.metrics.execution import ScopedMetricsContainer
 from apache_beam.pvalue import SideOutputValue
 from apache_beam.transforms import core
-from apache_beam.transforms import window
 from apache_beam.transforms.window import TimestampedValue
 from apache_beam.transforms.window import WindowFn
+from apache_beam.transforms.window import GlobalWindow
 from apache_beam.utils.windowed_value import WindowedValue
 
 
@@ -94,7 +94,7 @@ class DoFnRunner(Receiver):
 self.scoped_metrics_container = (scoped_metrics_container
  or ScopedMetricsContainer())
 
-global_window = window.GlobalWindow()
+global_window = GlobalWindow()
 
 # Need to support multiple iterations.
 side_inputs = list(side_inputs)
@@ -117,6 +117,11 @@ class DoFnRunner(Receiver):
 
 # TODO(Sourabhbajaj): Remove the usage of OldDoFn
 if isinstance(fn, core.NewDoFn):
+
+  class ArgPlaceholder(object):
+def __init__(self, placeholder):
+  self.placeholder = placeholder
+
   self.is_new_dofn = True
 
   # Stash values for use in new_dofn_process.
@@ -127,6 +132,70 @@ class DoFnRunner(Receiver):
   self.args = args if args else []
   self.kwargs = kwargs if kwargs else {}
   self.dofn = fn
+  self.dofn_process = fn.process
+
+  arguments, _, _, defaults = self.dofn.get_function_arguments('process')
+  defaults = defaults if defaults else []
+  self_in_args = int(self.dofn.is_process_bounded())
+
+  self.simple_process = (
+  not side_inputs and not args and not kwargs and not defaults)
+  if self.simple_process:
+return
+
+  # TODO(Sourabhbajaj) Rename this variable once oldDoFn is deprecated
+  self.has_windowed_side_inputs = (
+  self.has_windowed_side_inputs or
+  core.NewDoFn.WindowParam in defaults)
+
+  # Try to prepare all the arguments that can just be filled in
+  # without any additional work. in the process function.
+  # Also cache all the placeholders needed in the process function.
+
+  # Fill in sideInputs if they are globally windowed
+  if not self.has_windowed_side_inputs:
+self.args, self.kwargs = util.insert_values_in_args(
+args, kwargs, [si[global_window] for si in side_inputs])
+
+  # Create placeholder f

[GitHub] beam pull request #1909: Adding metrics() to DataflowPipelineResult.

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/1909

Adding metrics() to DataflowPipelineResult.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam add-dataflow-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1909


commit ff6512322ead8aed2216913e002da7d5a9793bf6
Author: Pablo 
Date:   2017-02-02T21:42:58Z

Adding metrics() to DataflowPipelineResult.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1380) Typo in programming guide

[Bill Neubauer (JIRA)]

Bill Neubauer created BEAM-1380:
---

 Summary: Typo in programming guide
 Key: BEAM-1380
 URL: https://issues.apache.org/jira/browse/BEAM-1380
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Bill Neubauer
Assignee: Davor Bonaci
Priority: Trivial


https://beam.apache.org/documentation/programming-guide/

s/perfroms/performs/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1905: Restores visibility of MetricMatchers.metricUpdate

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Closes #1905

[bchambers]

Repository: beam
Updated Branches:
  refs/heads/master fed80cec2 -> 70b16c74d


Closes #1905


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/70b16c74
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/70b16c74
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/70b16c74

Branch: refs/heads/master
Commit: 70b16c74dd97f746c3c08b16568a3066d9812a57
Parents: fed80ce 33052e8
Author: bchambers 
Authored: Thu Feb 2 12:38:55 2017 -0800
Committer: bchambers 
Committed: Thu Feb 2 12:38:55 2017 -0800

--
 .../apache/beam/sdk/metrics/MetricMatchers.java | 22 ++--
 1 file changed, 20 insertions(+), 2 deletions(-)
--

[2/2] beam git commit: Restores visibility of MetricMatchers.metricUpdate

[bchambers]

Restores visibility of MetricMatchers.metricUpdate

Documents that these functions can be used by runner tests,
and documents the couple of other matchers too.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/33052e8a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/33052e8a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/33052e8a

Branch: refs/heads/master
Commit: 33052e8a108fee4885332331e830e6179b3f067f
Parents: fed80ce
Author: Eugene Kirpichov 
Authored: Wed Feb 1 22:37:55 2017 -0800
Committer: bchambers 
Committed: Thu Feb 2 12:38:55 2017 -0800

--
 .../apache/beam/sdk/metrics/MetricMatchers.java | 22 ++--
 1 file changed, 20 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/33052e8a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
index b8109b3..3648c05 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricMatchers.java
@@ -29,7 +29,12 @@ import org.hamcrest.TypeSafeMatcher;
  */
 public class MetricMatchers {
 
-  static  Matcher> metricUpdate(final String name, final T 
update) {
+  /**
+   * Matches a {@link MetricUpdate} with the given name and contents.
+   *
+   * Visible since it may be used in runner-specific tests.
+   */
+  public static  Matcher> metricUpdate(final String name, 
final T update) {
 return new TypeSafeMatcher>() {
   @Override
   protected boolean matchesSafely(MetricUpdate item) {
@@ -47,7 +52,12 @@ public class MetricMatchers {
 };
   }
 
-  static  Matcher> metricUpdate(
+  /**
+   * Matches a {@link MetricUpdate} with the given namespace, name, step and 
contents.
+   *
+   * Visible since it may be used in runner-specific tests.
+   */
+  public static  Matcher> metricUpdate(
   final String namespace, final String name, final String step, final T 
update) {
 return new TypeSafeMatcher>() {
   @Override
@@ -70,6 +80,10 @@ public class MetricMatchers {
 };
   }
 
+  /**
+   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose attempted
+   * value equals the given value.
+   */
   public static  Matcher> attemptedMetricsResult(
   final String namespace, final String name, final String step, final T 
attempted) {
 return new TypeSafeMatcher>() {
@@ -108,6 +122,10 @@ public class MetricMatchers {
 };
   }
 
+  /**
+   * Matches a {@link MetricResult} with the given namespace, name and step, 
and whose committed
+   * value equals the given value.
+   */
   public static  Matcher> committedMetricsResult(
   final String namespace, final String name, final String step,
   final T committed) {

svn commit: r18089 - in /dev/beam/0.5.0/RC2: ./ apache-beam-0.5.0-source-release.zip apache-beam-0.5.0-source-release.zip.asc apache-beam-0.5.0-source-release.zip.md5 apache-beam-0.5.0-source-release.

[davor]

Author: davor
Date: Thu Feb  2 20:58:29 2017
New Revision: 18089

Log:
Add Apache Beam, version 0.5.0, release candidate #2.


Added:
dev/beam/0.5.0/RC2/
dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip   (with props)
dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.asc
dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.md5
dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.sha1

Added: dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip
==
Binary file - no diff available.

Propchange: dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.asc
==
--- dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.asc (added)
+++ dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.asc Thu Feb  2 
20:58:29 2017
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQEcBAABAgAGBQJYk4m0AAoJEMkEN+GPDTNP+vEH/1q/6bBBab+3hW9ZmZH4yymb
+65MLkWbteIvlJdIplD/eIUFlVgCDLj5HoJrjtsV081Z6pwZApaxIMj4qykjjYMPG
+KqSW9f4uyBJApqWCXiynsqn5XKS9uy+/AfSXQoLdcZ5Zm/AaMbaJfsxiVkKEsr8m
+r1hzQhOD7jSZJsq/ZFSzZ/3I7Y2EcGKhktovIiVscjt6gjsuGkuRvWZyBpHRpGcL
+puFsa2QGhpgRWcUyBWIKffWBbLRmaCMzpjjM1/O22SxeONZVrPvw/KjVmDZpVZrp
+eHv5pAN38F1RfpsaoaoCdvZYjs8aJvsxFf4Otfq7SL8Y6nai2N+9Agsj3CHjVx0=
+=3bl2
+-END PGP SIGNATURE-

Added: dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.md5
==
--- dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.md5 (added)
+++ dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.md5 Thu Feb  2 
20:58:29 2017
@@ -0,0 +1 @@
+a7afe87ac1ce5ac4ed47e44ec5314d8d  apache-beam-0.5.0-source-release.zip

Added: dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.sha1
==
--- dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.sha1 (added)
+++ dev/beam/0.5.0/RC2/apache-beam-0.5.0-source-release.zip.sha1 Thu Feb  2 
20:58:29 2017
@@ -0,0 +1 @@
+51b46967557e27d9af29761c7fc2d7a6c5ff4bff  apache-beam-0.5.0-source-release.zip

svn commit: r18088 - /dev/beam/0.4.0/

[davor]

Author: davor
Date: Thu Feb  2 20:55:45 2017
New Revision: 18088

Log:
Remove Apache Beam, version 0.4.0, from dev repository; it was previously 
released.


Removed:
dev/beam/0.4.0/

[GitHub] beam pull request #1904: Ignores errorprone annotations in enforcer config

[jkff]

Github user jkff closed the pull request at:

https://github.com/apache/beam/pull/1904


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.5.0-RC2

[davor]

Repository: beam
Updated Branches:
  refs/heads/release-0.5.0 4838ae16c -> 98274da06


[maven-release-plugin] rollback changes from release preparation of v0.5.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/98274da0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/98274da0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/98274da0

Branch: refs/heads/release-0.5.0
Commit: 98274da0637b051a0a991c339a83b90e5dc813f8
Parents: 4838ae1
Author: Davor Bonaci 
Authored: Thu Feb 2 11:45:59 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 11:45:59 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 35 files changed, 36 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/98274da0/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 8a5c2f9..e001d1c 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.5.0
+0.5.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/98274da0/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index f97ca0e..370d79f 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.5.0
+0.5.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/98274da0/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index aa251aa..4294c2d 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.5.0
+0.5.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/98274da0/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0be1f44..2281f67 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.5.0
+  0.5.0-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-v0.5.0-RC2
+release-0.5.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/98274da0/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 7fc787f..47139a6 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -22,7 +22,7 @@
   
  

[beam] Git Push Summary

[davor]

Repository: beam
Updated Tags:  refs/tags/v0.5.0-RC2 [created] cb4eb68a1

beam git commit: [maven-release-plugin] prepare release v0.5.0-RC2

[davor]

Repository: beam
Updated Branches:
  refs/heads/release-0.5.0 e7599c3e5 -> 4838ae16c


[maven-release-plugin] prepare release v0.5.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4838ae16
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4838ae16
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4838ae16

Branch: refs/heads/release-0.5.0
Commit: 4838ae16c172252bc0a15e3a984e085f82e25c2d
Parents: e7599c3
Author: Davor Bonaci 
Authored: Thu Feb 2 11:45:43 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 11:45:43 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 35 files changed, 36 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4838ae16/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index e001d1c..8a5c2f9 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.5.0-SNAPSHOT
+0.5.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/4838ae16/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 370d79f..f97ca0e 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.5.0-SNAPSHOT
+0.5.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/4838ae16/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 4294c2d..aa251aa 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.5.0-SNAPSHOT
+0.5.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/4838ae16/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 2281f67..0be1f44 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.5.0-SNAPSHOT
+  0.5.0
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-release-0.5.0
+v0.5.0-RC2
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/4838ae16/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 47139a6..7fc787f 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-

[GitHub] beam pull request #1908: Invoke onMerge in AfterWatermarkEarly

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1908


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1908

[davor]

This closes #1908


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e7599c3e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e7599c3e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e7599c3e

Branch: refs/heads/release-0.5.0
Commit: e7599c3e5e97338c9b7a4c07940e551fbe506561
Parents: 0d0d497 69165f8
Author: Davor Bonaci 
Authored: Thu Feb 2 11:30:40 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 2 11:30:40 2017 -0800

--
 .../triggers/AfterWatermarkStateMachine.java |  1 +
 .../triggers/AfterWatermarkStateMachineTest.java | 19 +++
 2 files changed, 20 insertions(+)
--

[1/2] beam git commit: Invoke onMerge in AfterWatermarkEarly

[davor]

Repository: beam
Updated Branches:
  refs/heads/release-0.5.0 0d0d49705 -> e7599c3e5


Invoke onMerge in AfterWatermarkEarly

This ensures that any triggering state manipulations appropriately
notify the early subtrigger before resetting the finished bit. This
ensures that any timers or state is appropriately migrated to the
merged window.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69165f85
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69165f85
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69165f85

Branch: refs/heads/release-0.5.0
Commit: 69165f853d0c434ecd46ae445eb98894b2029360
Parents: 0d0d497
Author: Thomas Groh 
Authored: Thu Feb 2 09:23:45 2017 -0800
Committer: Thomas Groh 
Committed: Thu Feb 2 11:07:22 2017 -0800

--
 .../triggers/AfterWatermarkStateMachine.java |  1 +
 .../triggers/AfterWatermarkStateMachineTest.java | 19 +++
 2 files changed, 20 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/69165f85/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
index 524c057..e83c2f8 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
@@ -129,6 +129,7 @@ public class AfterWatermarkStateMachine {
   // the new merged window, because even if the merged window is "done" 
some pending elements
   // haven't had a chance to fire.
   if (!earlyContext.trigger().finishedInAllMergingWindows() || 
!endOfWindowReached(c)) {
+earlySubtrigger.invokeOnMerge(earlyContext);
 earlyContext.trigger().setFinished(false);
 if (lateTrigger != null) {
   ExecutableTriggerStateMachine lateSubtrigger = 
c.trigger().subTrigger(LATE_INDEX);

http://git-wip-us.apache.org/repos/asf/beam/blob/69165f85/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
index 119c937..e4d10a0 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
@@ -21,8 +21,10 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
+import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnMergeContext;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachineTester.SimpleTriggerStateMachineTester;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -293,6 +295,23 @@ public class AfterWatermarkStateMachineTest {
 assertTrue(tester.shouldFire(mergedWindow));
   }
 
+  @Test
+  public void testEarlyAndLateOnMergeSubtriggerMerges() throws Exception {
+tester =
+TriggerStateMachineTester.forTrigger(
+AfterWatermarkStateMachine.pastEndOfWindow()
+.withEarlyFirings(mockEarly)
+.withLateFirings(mockLate),
+Sessions.withGapDuration(Duration.millis(10)));
+
+tester.injectElements(1);
+tester.injectElements(5);
+
+// Merging should re-activate the early trigger in the merged window
+tester.mergeWindows();
+verify(mockEarly).onMerge(Mockito.any(OnMergeContext.class));
+  }
+
   /**
* Tests that the trigger rewinds to be non-finished in the merged window.
*

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2511

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2510

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1908: Invoke onMerge in AfterWatermarkEarly

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1908

Invoke onMerge in AfterWatermarkEarly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This ensures that any triggering state manipulations appropriately
notify the early subtrigger before resetting the finished bit. This
ensures that any timers or state is appropriately migrated to the
merged window.

Merged to master in #1907 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
fix_AfterWatermark_merging_cherry-pick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1908.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1908


commit 69165f853d0c434ecd46ae445eb98894b2029360
Author: Thomas Groh 
Date:   2017-02-02T17:23:45Z

Invoke onMerge in AfterWatermarkEarly

This ensures that any triggering state manipulations appropriately
notify the early subtrigger before resetting the finished bit. This
ensures that any timers or state is appropriately migrated to the
merged window.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #144: Programming guide update

[hadarhg]

GitHub user hadarhg reopened a pull request:

https://github.com/apache/beam-site/pull/144

Programming guide update

- Added data encoding section for Java and Python
- Changed some Python code snippets to use include tags 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site prog-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #144


commit c05c530236a115828212c04e4f76f79cb7eca5e1
Author: Hadar Hod 
Date:   2017-02-01T20:19:58Z

Added Python snippets. Merged changes.

commit a9e9524c862c86fbbffc1fa75dc0452778f0ec9a
Author: Hadar Hod 
Date:   2017-02-02T18:43:47Z

Added data encoding to programming guide.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #144: Programming guide update

[hadarhg]

Github user hadarhg closed the pull request at:

https://github.com/apache/beam-site/pull/144


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #144: Programming guide update

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/beam-site/pull/144

Programming guide update

- Added data encoding section for Java and Python
- Changed some Python code snippets to use include tags 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site prog-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #144


commit c05c530236a115828212c04e4f76f79cb7eca5e1
Author: Hadar Hod 
Date:   2017-02-01T20:19:58Z

Added Python snippets. Merged changes.

commit a9e9524c862c86fbbffc1fa75dc0452778f0ec9a
Author: Hadar Hod 
Date:   2017-02-02T18:43:47Z

Added data encoding to programming guide.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1319) PipelineOptions subclasses defined in the main session could be duplicated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850300#comment-15850300
 ] 

ASF GitHub Bot commented on BEAM-1319:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1889


> PipelineOptions subclasses defined in the main session could be duplicated
> --
>
> Key: BEAM-1319
> URL: https://issues.apache.org/jira/browse/BEAM-1319
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>
> Duplication is caused as a result of the save_main_session option.
> This also breaks argparse because same options will be defined multiple times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1889: [BEAM-1319] Add conflict resolution to the Pipeline...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1889


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1889

[altay]

This closes #1889


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fed80cec
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fed80cec
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fed80cec

Branch: refs/heads/master
Commit: fed80cec20c27370e024dcf23560e5d96957925d
Parents: 0a901d9 2ce7784
Author: Ahmet Altay 
Authored: Thu Feb 2 10:30:32 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 10:30:32 2017 -0800

--
 sdks/python/apache_beam/utils/pipeline_options.py  |  7 +++
 .../apache_beam/utils/pipeline_options_test.py | 17 +
 2 files changed, 24 insertions(+)
--

[1/2] beam git commit: In some instances where a PipelineOptions subclass was defined in the main session and save_main_session option is enabled, that subclass may appear multiple times in the Pipeli

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 0a901d948 -> fed80cec2


In some instances where a PipelineOptions subclass was defined in the
main session and save_main_session option is enabled, that subclass may
appear multiple times in the PipelineOptions.__subclassess__() list.
This is causing problems with the argparse because options are not
unique any more.

This changes filter the subclasses by name, and pick the last unique
instance of each subclass.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2ce77847
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2ce77847
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2ce77847

Branch: refs/heads/master
Commit: 2ce7784731a0a2e4daaaefbc1bb475c0f2a575d4
Parents: 0a901d9
Author: Ahmet Altay 
Authored: Wed Feb 1 10:39:59 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Feb 2 10:30:20 2017 -0800

--
 sdks/python/apache_beam/utils/pipeline_options.py  |  7 +++
 .../apache_beam/utils/pipeline_options_test.py | 17 +
 2 files changed, 24 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2ce77847/sdks/python/apache_beam/utils/pipeline_options.py
--
diff --git a/sdks/python/apache_beam/utils/pipeline_options.py 
b/sdks/python/apache_beam/utils/pipeline_options.py
index f7d7597..7840eb7 100644
--- a/sdks/python/apache_beam/utils/pipeline_options.py
+++ b/sdks/python/apache_beam/utils/pipeline_options.py
@@ -116,9 +116,16 @@ class PipelineOptions(HasDisplayData):
 Returns:
   Dictionary of all args and values.
 """
+
+# TODO(BEAM-1319): PipelineOption sub-classes in the main session might be
+# repeated. Pick last unique instance of each subclass to avoid conflicts.
 parser = argparse.ArgumentParser()
+subset = {}
 for cls in PipelineOptions.__subclasses__():
+  subset[str(cls)] = cls
+for cls in subset.values():
   cls._add_argparse_args(parser)  # pylint: disable=protected-access
+
 known_args, _ = parser.parse_known_args(self._flags)
 result = vars(known_args)
 

http://git-wip-us.apache.org/repos/asf/beam/blob/2ce77847/sdks/python/apache_beam/utils/pipeline_options_test.py
--
diff --git a/sdks/python/apache_beam/utils/pipeline_options_test.py 
b/sdks/python/apache_beam/utils/pipeline_options_test.py
index 054b6a5..507a827 100644
--- a/sdks/python/apache_beam/utils/pipeline_options_test.py
+++ b/sdks/python/apache_beam/utils/pipeline_options_test.py
@@ -170,6 +170,23 @@ class PipelineOptionsTest(unittest.TestCase):
 options = PipelineOptions(flags=[''])
 self.assertEqual(options.get_all_options()['template_location'], None)
 
+  def test_redefine_options(self):
+
+class TestRedefinedOptios(PipelineOptions):  # pylint: 
disable=unused-variable
+
+  @classmethod
+  def _add_argparse_args(cls, parser):
+parser.add_argument('--redefined_flag', action='store_true')
+
+class TestRedefinedOptios(PipelineOptions):
+
+  @classmethod
+  def _add_argparse_args(cls, parser):
+parser.add_argument('--redefined_flag', action='store_true')
+
+options = PipelineOptions(['--redefined_flag'])
+self.assertEqual(options.get_all_options()['redefined_flag'], True)
+
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
   unittest.main()

[jira] [Closed] (BEAM-1378) Late data dropping was removed from Flink Runner

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-1378.
-

> Late data dropping was removed from Flink Runner
> 
>
> Key: BEAM-1378
> URL: https://issues.apache.org/jira/browse/BEAM-1378
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1378) Late data dropping was removed from Flink Runner

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1378.
---
   Resolution: Fixed
Fix Version/s: (was: 0.5.0)
   Not applicable

> Late data dropping was removed from Flink Runner
> 
>
> Key: BEAM-1378
> URL: https://issues.apache.org/jira/browse/BEAM-1378
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1370) AfterWatermarkEarlyAndLate does not invoke the onMerge of the early trigger

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850280#comment-15850280
 ] 

ASF GitHub Bot commented on BEAM-1370:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1907


> AfterWatermarkEarlyAndLate does not invoke the onMerge of the early trigger
> ---
>
> Key: BEAM-1370
> URL: https://issues.apache.org/jira/browse/BEAM-1370
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> When a window merges in AfterWatermarkStateMachine, if the early trigger is 
> not finished the state machine should invoke its onMerge. This ensures, for 
> example, that with merging windows with a processing time early trigger, that 
> trigger is propagated to the merged windows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1378) Late data dropping was removed from Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850279#comment-15850279
 ] 

ASF GitHub Bot commented on BEAM-1378:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1903


> Late data dropping was removed from Flink Runner
> 
>
> Key: BEAM-1378
> URL: https://issues.apache.org/jira/browse/BEAM-1378
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1907: [BEAM-1370] Invoke onMerge in AfterWatermarkEarly

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1907


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1265) Add streaming support to DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850278#comment-15850278
 ] 

ASF GitHub Bot commented on BEAM-1265:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1899


> Add streaming support to DirectRunner
> -
>
> Key: BEAM-1265
> URL: https://issues.apache.org/jira/browse/BEAM-1265
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>
> Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Invoke onMerge in AfterWatermarkEarly

[tgroh]

Invoke onMerge in AfterWatermarkEarly

This ensures that any triggering state manipulations appropriately
notify the early subtrigger before resetting the finished bit. This
ensures that any timers or state is appropriately migrated to the
merged window.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/13728c1b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/13728c1b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/13728c1b

Branch: refs/heads/master
Commit: 13728c1b940ece4474708dc1a3989dfb4b4d86e3
Parents: 311425c
Author: Thomas Groh 
Authored: Thu Feb 2 09:23:45 2017 -0800
Committer: Thomas Groh 
Committed: Thu Feb 2 10:27:17 2017 -0800

--
 .../triggers/AfterWatermarkStateMachine.java |  1 +
 .../triggers/AfterWatermarkStateMachineTest.java | 19 +++
 2 files changed, 20 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/13728c1b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
index 524c057..e83c2f8 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
@@ -129,6 +129,7 @@ public class AfterWatermarkStateMachine {
   // the new merged window, because even if the merged window is "done" 
some pending elements
   // haven't had a chance to fire.
   if (!earlyContext.trigger().finishedInAllMergingWindows() || 
!endOfWindowReached(c)) {
+earlySubtrigger.invokeOnMerge(earlyContext);
 earlyContext.trigger().setFinished(false);
 if (lateTrigger != null) {
   ExecutableTriggerStateMachine lateSubtrigger = 
c.trigger().subTrigger(LATE_INDEX);

http://git-wip-us.apache.org/repos/asf/beam/blob/13728c1b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
index 119c937..e4d10a0 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
@@ -21,8 +21,10 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
+import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnMergeContext;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachineTester.SimpleTriggerStateMachineTester;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -293,6 +295,23 @@ public class AfterWatermarkStateMachineTest {
 assertTrue(tester.shouldFire(mergedWindow));
   }
 
+  @Test
+  public void testEarlyAndLateOnMergeSubtriggerMerges() throws Exception {
+tester =
+TriggerStateMachineTester.forTrigger(
+AfterWatermarkStateMachine.pastEndOfWindow()
+.withEarlyFirings(mockEarly)
+.withLateFirings(mockLate),
+Sessions.withGapDuration(Duration.millis(10)));
+
+tester.injectElements(1);
+tester.injectElements(5);
+
+// Merging should re-activate the early trigger in the merged window
+tester.mergeWindows();
+verify(mockEarly).onMerge(Mockito.any(OnMergeContext.class));
+  }
+
   /**
* Tests that the trigger rewinds to be non-finished in the merged window.
*

[1/2] beam git commit: This closes #1907

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 311425ced -> 0a901d948


This closes #1907


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a901d94
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a901d94
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a901d94

Branch: refs/heads/master
Commit: 0a901d9489afec98c4b0d6db02e4837030e2a404
Parents: 311425c 13728c1
Author: Thomas Groh 
Authored: Thu Feb 2 10:27:17 2017 -0800
Committer: Thomas Groh 
Committed: Thu Feb 2 10:27:17 2017 -0800

--
 .../triggers/AfterWatermarkStateMachine.java |  1 +
 .../triggers/AfterWatermarkStateMachineTest.java | 19 +++
 2 files changed, 20 insertions(+)
--

[GitHub] beam pull request #1903: [BEAM-1378] Release-0.5.0 cherry pick: Drop late da...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1903


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1899: [BEAM-1265] Fail if --streaming option is used

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1899


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---