[GitHub] incubator-beam pull request #1645: Add properties to skip tests for particul...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1645

Add properties to skip tests for particular modules

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

A discussion piece. Used pervasively, we might trim a lot of time and also 
improve usability.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam skip-all-the-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1645.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1645


commit b5dea398cf14fe8717fa27ad166fa860c55ab7d1
Author: Kenneth Knowles 
Date:   2016-12-16T06:55:20Z

Add properties to skip tests for particular modules




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1642: Delete deprecated TimerCallback

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1642


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753479#comment-15753479
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1644

[BEAM-362] Port runners to runners-core AggregatoryFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

For context: `runners.core.AggregatorFactory` is a copy of 
`sdk.transforms.Aggregator.AggregatorFactory`. So I am just porting everything 
to the non-deprecated bit so I can delete it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
runners-core-AggregatorFactory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1644


commit 63ac16669e7884dd42d431b9948ed675f7af3f03
Author: Kenneth Knowles 
Date:   2016-12-16T05:06:14Z

Port runners to runners-core AggregatoryFactory




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1644: [BEAM-362] Port runners to runners-core A...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1644

[BEAM-362] Port runners to runners-core AggregatoryFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

For context: `runners.core.AggregatorFactory` is a copy of 
`sdk.transforms.Aggregator.AggregatorFactory`. So I am just porting everything 
to the non-deprecated bit so I can delete it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
runners-core-AggregatorFactory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1644


commit 63ac16669e7884dd42d431b9948ed675f7af3f03
Author: Kenneth Knowles 
Date:   2016-12-16T05:06:14Z

Port runners to runners-core AggregatoryFactory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1641: Remove deprecated AggregatorFactory from ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1641


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753444#comment-15753444
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1643

[BEAM-362] Move InMemoryTimerInternals to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff (I'll grab a committer later, but your contributions are thing 
affected)

The only substantive aspect of this change is the move of the timer 
internals directly into `ProcessFn`. I think this is right or at least the 
right compromise for many reasons.

 - `TimerInternals` is really runner-facing; we don't want that interface 
in the SDK.
 - Changes to `TimerInternals` incur worker compatibility concerns, so 
getting it into runners-core is a win. (I have to change it soon, so I am 
trying to make my life easier)
 - `DoFnTester` doesn't actually support timers at all, so it didn't make 
sense for them to be in there.
 - When `DoFnTester` does support timers, it is trivial to make its own 
priority queues, and it will also want to offer greater insight via a richer 
`TestingTimerInternals` sort of implementation, which needs not implement the 
same interface, etc.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
InMemoryTimerInternals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1643.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1643


commit 5d0bf9895654c12e50410a98347eb9177de00b1d
Author: Kenneth Knowles 
Date:   2016-12-16T04:45:56Z

Move InMemoryTimerInternals to runners-core




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1643: [BEAM-362] Move InMemoryTimerInternals to...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1643

[BEAM-362] Move InMemoryTimerInternals to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff (I'll grab a committer later, but your contributions are thing 
affected)

The only substantive aspect of this change is the move of the timer 
internals directly into `ProcessFn`. I think this is right or at least the 
right compromise for many reasons.

 - `TimerInternals` is really runner-facing; we don't want that interface 
in the SDK.
 - Changes to `TimerInternals` incur worker compatibility concerns, so 
getting it into runners-core is a win. (I have to change it soon, so I am 
trying to make my life easier)
 - `DoFnTester` doesn't actually support timers at all, so it didn't make 
sense for them to be in there.
 - When `DoFnTester` does support timers, it is trivial to make its own 
priority queues, and it will also want to offer greater insight via a richer 
`TestingTimerInternals` sort of implementation, which needs not implement the 
same interface, etc.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
InMemoryTimerInternals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1643.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1643


commit 5d0bf9895654c12e50410a98347eb9177de00b1d
Author: Kenneth Knowles 
Date:   2016-12-16T04:45:56Z

Move InMemoryTimerInternals to runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1642: Delete deprecated TimerCallback

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1642

Delete deprecated TimerCallback

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tweise I missed doing this completely, because I was focused on just a 
couple annoying methods. But now there really are no `@Deprecated` and 
`TimerCallback` is deleted.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam TimerCallback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1642.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1642


commit 33a977a7a157f4f8dd3d2abc8facdbaa6a723143
Author: Kenneth Knowles 
Date:   2016-12-16T04:26:40Z

Delete deprecated TimerCallback




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1641: Remove deprecated AggregatorFactory from ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1641

Remove deprecated AggregatorFactory from SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tgroh easy one

Analogous Dataflow worker changes are en route, not requiring a dance, but 
I thought I'd just open this since they won't affect the code review. This is a 
trivial port from one copy to another of this interface.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam AggregatorFactory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1641.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1641


commit baa3e4997e061eed69d33fd1ffbfd2af832bf0f6
Author: Kenneth Knowles 
Date:   2016-12-16T04:13:25Z

Remove deprecated AggregatorFactory from SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1640: Show timestamps on log lines in Jenkins

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1640

Show timestamps on log lines in Jenkins

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @dhalperi This adds timestamp prefixes to Maven log messages. Unsure if 
we can just toggle this at the Jenkins level, which would be superior.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam jenkins-timestamps

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1640.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1640


commit e4179994549f6bcdd93f513ea21bac4551d07019
Author: Kenneth Knowles 
Date:   2016-12-16T03:51:22Z

Show timestamps on log lines in Jenkins




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1639: [BEAM-1112] Python E2E Test Framework And...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1639

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - E2e test framework that supports TestRunner start and verify pipeline 
job.
   - add `TestOptions` which defined `on_success_matcher` that is used to 
verify state/output of pipeline job.
   - validate `on_success_matcher` before pipeline execution to make sure 
it's unpicklable to a subclass of BaseMatcher.
   - create a `TestDataflowRunner` which provide functionalities of 
`DataflowRunner` plus result verification.
   - provide a test verifier `PipelineStateMatcher` that can verify 
pipeline job finished in DONE or not.
 - Add wordcount_it (it = integration test) that build e2e test based on 
existing wordcount pipeline.
   - include wordcount_it to nose collector, so that wordcount_it can be 
collected and run by nose.
   - skip ITs when running unit tests from tox in precommit and postcommit.

Current changes will not change behavior of existing pre/postcommit.
Test is done by running `tox -e py27 -c sdks/python/tox.ini` for unit test 
and running wordcount_it with `TestDataflowRunner` on service 
([link](https://pantheon.corp.google.com/dataflow/job/2016-12-15_17_36_16-3857167705491723621?project=google.com:clouddfe)).

TODO:
 - Output data verifier that verify pipeline output that stores in 
filesystem.
 - Add wordcount_it to precommit and replace existing wordcount execution 
command in postcommit with a better structured nose command.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam e2e-testrunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639


commit e1e1fa3a60e1fe234829432d144d6689e240b6f0
Author: Mark Liu 
Date:   2016-12-16T01:41:20Z

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

commit 0e7007879ee082e3afe5db36107f51c03274f3f5
Author: Mark Liu 
Date:   2016-12-16T02:55:53Z

fixup! Fix Code Style




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1112) Python E2E Integration Test Framework - Batch Only

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753301#comment-15753301
 ] 

ASF GitHub Bot commented on BEAM-1112:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1639

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - E2e test framework that supports TestRunner start and verify pipeline 
job.
   - add `TestOptions` which defined `on_success_matcher` that is used to 
verify state/output of pipeline job.
   - validate `on_success_matcher` before pipeline execution to make sure 
it's unpicklable to a subclass of BaseMatcher.
   - create a `TestDataflowRunner` which provide functionalities of 
`DataflowRunner` plus result verification.
   - provide a test verifier `PipelineStateMatcher` that can verify 
pipeline job finished in DONE or not.
 - Add wordcount_it (it = integration test) that build e2e test based on 
existing wordcount pipeline.
   - include wordcount_it to nose collector, so that wordcount_it can be 
collected and run by nose.
   - skip ITs when running unit tests from tox in precommit and postcommit.

Current changes will not change behavior of existing pre/postcommit.
Test is done by running `tox -e py27 -c sdks/python/tox.ini` for unit test 
and running wordcount_it with `TestDataflowRunner` on service 
([link](https://pantheon.corp.google.com/dataflow/job/2016-12-15_17_36_16-3857167705491723621?project=google.com:clouddfe)).

TODO:
 - Output data verifier that verify pipeline output that stores in 
filesystem.
 - Add wordcount_it to precommit and replace existing wordcount execution 
command in postcommit with a better structured nose command.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam e2e-testrunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639


commit e1e1fa3a60e1fe234829432d144d6689e240b6f0
Author: Mark Liu 
Date:   2016-12-16T01:41:20Z

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

commit 0e7007879ee082e3afe5db36107f51c03274f3f5
Author: Mark Liu 
Date:   2016-12-16T02:55:53Z

fixup! Fix Code Style




> Python E2E Integration Test Framework - Batch Only
> --
>
> Key: BEAM-1112
> URL: https://issues.apache.org/jira/browse/BEAM-1112
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Parity with Java. 
> Build e2e integration test framework that can configure and run batch 
> pipeline with specified test runner, wait for pipeline execution and verify 
> results with given verifiers in the end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-1125) Rename PTransform.apply to PTransform.expand

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay closed BEAM-1125.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Rename PTransform.apply to PTransform.expand
> 
>
> Key: BEAM-1125
> URL: https://issues.apache.org/jira/browse/BEAM-1125
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
> Fix For: Not applicable
>
>
> For context see:
> [BEAM-438] https://issues.apache.org/jira/browse/BEAM-438
> [PR #1538] https://github.com/apache/incubator-beam/pull/1538
> https://lists.apache.org/thread.html/b4d9bcfbfeaa5dbcd5b68fd2344cdffe45587ff88cb714638504e759@%3Cdev.beam.apache.org%3E
> This requires renaming the apply method, updating all custom PTransforms, and 
> runners where transform.apply is called. (Based on the Java PR, this could be 
> easily done with a refactoring tool.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1634: [BEAM-1125] Rename PTransform.apply() to ...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1634


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1635: Remove deprecated methods of InMemoryTime...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1635


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1635

[kenn]

This closes #1635

  Remove deprecated methods of InMemoryTimerInternals


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d624d3b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d624d3b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d624d3b6

Branch: refs/heads/master
Commit: d624d3b6bb3d3b68affac0906b97aa1f33e57ddd
Parents: 8c18913 e26f407
Author: Kenneth Knowles 
Authored: Thu Dec 15 19:00:45 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 19:00:45 2016 -0800

--
 .../sdk/util/state/InMemoryTimerInternals.java  | 26 -
 .../util/state/InMemoryTimerInternalsTest.java  | 40 
 2 files changed, 15 insertions(+), 51 deletions(-)
--

[1/2] incubator-beam git commit: Remove deprecated methods of InMemoryTimerInternals

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8c1891375 -> d624d3b6b


Remove deprecated methods of InMemoryTimerInternals


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e26f4075
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e26f4075
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e26f4075

Branch: refs/heads/master
Commit: e26f4075af6f2c990e23dc9f8fc8be2233652a9f
Parents: 5a3ace4
Author: Kenneth Knowles 
Authored: Thu Dec 15 16:02:23 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 16:02:23 2016 -0800

--
 .../sdk/util/state/InMemoryTimerInternals.java  | 26 -
 .../util/state/InMemoryTimerInternalsTest.java  | 40 
 2 files changed, 15 insertions(+), 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26f4075/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
index 159b583..44f9016 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
@@ -273,17 +273,6 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 }
   }
 
-  /** Advances input watermark to the given value and fires event-time timers 
accordingly.
-   *
-   *  @deprecated Use advanceInputWatermark without callback and 
fireEventTimers.
-   */
-  @Deprecated
-  public void advanceInputWatermark(
-  TimerCallback timerCallback, Instant newInputWatermark) throws Exception 
{
-advanceInputWatermark(newInputWatermark);
-advanceAndFire(timerCallback, newInputWatermark, TimeDomain.EVENT_TIME);
-  }
-
   /** Advances processing time to the given value and fires processing-time 
timers accordingly.
*
*  @deprecated Use advanceProcessingTime without callback and 
fireProcessingTimers.
@@ -295,21 +284,6 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 advanceAndFire(timerCallback, newProcessingTime, 
TimeDomain.PROCESSING_TIME);
   }
 
-  /**
-   * Advances synchronized processing time to the given value and fires 
processing-time timers
-   * accordingly.
-   *
-   *  @deprecated Use advanceInputWatermark without callback and 
fireSynchronizedProcessingTimers.
-   */
-  @Deprecated
-  public void advanceSynchronizedProcessingTime(
-  TimerCallback timerCallback, Instant newSynchronizedProcessingTime)
-  throws Exception {
-advanceSynchronizedProcessingTime(newSynchronizedProcessingTime);
-advanceAndFire(
-timerCallback, newSynchronizedProcessingTime, 
TimeDomain.SYNCHRONIZED_PROCESSING_TIME);
-  }
-
   @Deprecated
   private void advanceAndFire(
   TimerCallback timerCallback, Instant currentTime, TimeDomain domain)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e26f4075/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
index 1e42864..4a2763c 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/InMemoryTimerInternalsTest.java
@@ -17,20 +17,18 @@
  */
 package org.apache.beam.sdk.util.state;
 
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
-import static org.mockito.Mockito.times;
+import static org.junit.Assert.assertThat;
 
 import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.joda.time.Instant;
-import org.junit.Before;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.MockitoAnnotations;
 
 /**
  * Tests for {@link InMemoryTimerInternals}.
@@ -40,14 +38,6 @@ public class InMemoryTimerInternalsTest {
 
   private static final StateNamespace NS1 = new StateNamespaceForTest("NS1");
 
-  @Mock
-  private TimerCallback timerCallback;
-
-  @Before
-  public void setUp() {
-

[GitHub] incubator-beam pull request #1638: Make a RunnableOnService test for Metrics

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1638

Make a RunnableOnService test for Metrics

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam ros-metrics-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1638.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1638


commit 6f57c0bda89b49576a09d2babe386632522b8544
Author: bchambers 
Date:   2016-12-16T01:04:59Z

Add UsesMetrics interface and exclude from runners

commit 850af9c5e2566e1127ab8e5c0e2f1b50e3cc404e
Author: bchambers 
Date:   2016-12-16T01:04:21Z

Add Serializability as needed for Metrics

commit 960f931b6e1893b85feda157e11b354480f9e62b
Author: bchambers 
Date:   2016-12-16T01:05:16Z

Add runnable on service test for Metrics

commit 4d698cec4b8353d1e7cdc6bcc6b762d59874035e
Author: bchambers 
Date:   2016-12-16T01:48:23Z

fixup! Remove test from DirectRunnerTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2116

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1084) Update apitools to version 0.5.6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753081#comment-15753081
 ] 

ASF GitHub Bot commented on BEAM-1084:
--

Github user sb2nov closed the pull request at:

https://github.com/apache/incubator-beam/pull/1501


> Update apitools to version 0.5.6
> 
>
> Key: BEAM-1084
> URL: https://issues.apache.org/jira/browse/BEAM-1084
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>
> There are some fixes to JsonValue that should be included in beam 
> (https://github.com/google/apitools/pull/136) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1501: [BEAM-1084] Update Apitools to version 0....

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/incubator-beam/pull/1501


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Update Apitools to version 0.5.6

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e26527873 -> 3b4fd5c7d


Update Apitools to version 0.5.6

This brings in the fix to https://github.com/google/apitools/pull/136
needed for the BigQuery reader.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/63074312
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/63074312
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/63074312

Branch: refs/heads/python-sdk
Commit: 63074312aeb44b5db7f4e914c64864483f6a6510
Parents: e265278
Author: Sourabh Bajaj 
Authored: Sat Dec 3 09:02:32 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 17:11:58 2016 -0800

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/63074312/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 033afc7..f6357b6 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -86,7 +86,7 @@ else:
 REQUIRED_PACKAGES = [
 'avro>=1.7.7,<2.0.0',
 'dill>=0.2.5,<0.3',
-'google-apitools>=0.5.2,<1.0.0',
+'google-apitools>=0.5.6,<1.0.0',
 'googledatastore==6.4.1',
 'httplib2>=0.8,<0.10',
 'mock>=1.0.1,<3.0.0',

[2/2] incubator-beam git commit: Closes #1501

[robertwb]

Closes #1501


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b4fd5c7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3b4fd5c7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3b4fd5c7

Branch: refs/heads/python-sdk
Commit: 3b4fd5c7d962987405dc157e6b84788af61f6413
Parents: e265278 6307431
Author: Robert Bradshaw 
Authored: Thu Dec 15 17:12:55 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 17:12:55 2016 -0800

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Resolved] (BEAM-1104) WordCount: Metrics error in the DirectRunner

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers resolved BEAM-1104.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> WordCount: Metrics error in the DirectRunner
> 
>
> Key: BEAM-1104
> URL: https://issues.apache.org/jira/browse/BEAM-1104
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Daniel Halperin
>Assignee: Ben Chambers
> Fix For: 0.4.0-incubating
>
>
> I'm following the Beam quickstart to analyze the pom.xml for the examples 
> archetype in the DirectRunner:
> Generate the project:
> {code}
> mvn archetype:generate \
>   
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots 
> \  
>   -DarchetypeGroupId=org.apache.beam \
>   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
>   -DarchetypeVersion=LATEST \
>   -DgroupId=org.example \
>   -DartifactId=word-count-beam \
>   -Dversion="0.1" \
>   -Dpackage=org.apache.beam.examples \
>   -DinteractiveMode=false
> {code}
> Count words in the pom.xml:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--inputFile=pom.xml --output=direct/counts" -Pdirect-runner
> {code}
> The logs:
> {code}
> INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.io.FileBasedSource 
> expandFilePattern
> INFO: Matched 1 files for pattern pom.xml
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.metrics.MetricsEnvironment 
> getCurrentContainer
> SEVERE: Unable to update metrics on the current thread. Most likely caused by 
> using metrics outside the managed work-execution thread.
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.io.Write$Bound$1 processElement
> INFO: Initializing write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@26bbd1cf
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$2 processElement
> INFO: Finalizing write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@3701012a.
> {code}
> Presumably, this {{SEVERE}} warning is indicative of a bug (or should be 
> masked).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC3

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 3d67e7a1f -> 93045e9c1


[maven-release-plugin] rollback changes from release preparation of 
v0.4.0-incubating-RC3


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/93045e9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/93045e9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/93045e9c

Branch: refs/heads/release-0.4.0-incubating
Commit: 93045e9c1b99b540bed2b3d93d568fd18ec9423e
Parents: 3d67e7a
Author: Davor Bonaci 
Authored: Thu Dec 15 17:01:44 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 17:01:44 2016 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 7333c10..e541edb 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 59d8f4d..8591955 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 65c6f1b..eb42861 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 6d7da5d..e860c7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.4.0-incubating
+  0.4.0-incubating-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-v0.4.0-incubating-RC3
+release-0.4.0-incubating
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/runners/apex/pom.xml

incubator-beam git commit: [maven-release-plugin] prepare release v0.4.0-incubating-RC3

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating bce0a0d50 -> 3d67e7a1f


[maven-release-plugin] prepare release v0.4.0-incubating-RC3


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3d67e7a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3d67e7a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3d67e7a1

Branch: refs/heads/release-0.4.0-incubating
Commit: 3d67e7a1fd2f988ca286aa6b8a8d927b986848de
Parents: bce0a0d
Author: Davor Bonaci 
Authored: Thu Dec 15 17:01:29 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 17:01:29 2016 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index e541edb..7333c10 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 8591955..59d8f4d 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index eb42861..65c6f1b 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e860c7a..6d7da5d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.4.0-incubating-SNAPSHOT
+  0.4.0-incubating
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-release-0.4.0-incubating
+v0.4.0-incubating-RC3
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/runners/apex/pom.xml

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.0-incubating-RC3 [created] 112e38e4a

[jira] [Commented] (BEAM-1104) WordCount: Metrics error in the DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753030#comment-15753030
 ] 

ASF GitHub Bot commented on BEAM-1104:
--

Github user bjchambers closed the pull request at:

https://github.com/apache/incubator-beam/pull/1615


> WordCount: Metrics error in the DirectRunner
> 
>
> Key: BEAM-1104
> URL: https://issues.apache.org/jira/browse/BEAM-1104
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Daniel Halperin
>Assignee: Ben Chambers
>
> I'm following the Beam quickstart to analyze the pom.xml for the examples 
> archetype in the DirectRunner:
> Generate the project:
> {code}
> mvn archetype:generate \
>   
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots 
> \  
>   -DarchetypeGroupId=org.apache.beam \
>   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
>   -DarchetypeVersion=LATEST \
>   -DgroupId=org.example \
>   -DartifactId=word-count-beam \
>   -Dversion="0.1" \
>   -Dpackage=org.apache.beam.examples \
>   -DinteractiveMode=false
> {code}
> Count words in the pom.xml:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--inputFile=pom.xml --output=direct/counts" -Pdirect-runner
> {code}
> The logs:
> {code}
> INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.io.FileBasedSource 
> expandFilePattern
> INFO: Matched 1 files for pattern pom.xml
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.metrics.MetricsEnvironment 
> getCurrentContainer
> SEVERE: Unable to update metrics on the current thread. Most likely caused by 
> using metrics outside the managed work-execution thread.
> Dec 07, 2016 9:42:03 PM org.apache.beam.sdk.io.Write$Bound$1 processElement
> INFO: Initializing write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@26bbd1cf
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$WriteBundles 
> processElement
> INFO: Opening writer for write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@19371061
> Dec 07, 2016 9:42:04 PM org.apache.beam.sdk.io.Write$Bound$2 processElement
> INFO: Finalizing write operation 
> org.apache.beam.sdk.io.TextIO$TextSink$TextWriteOperation@3701012a.
> {code}
> Presumably, this {{SEVERE}} warning is indicative of a bug (or should be 
> masked).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1615: [BEAM-1104] Don't incorrectly log error i...

[bjchambers]

Github user bjchambers closed the pull request at:

https://github.com/apache/incubator-beam/pull/1615


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Rename PTransform.apply() to PTransform.expand()

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk d3c887480 -> e26527873


Rename PTransform.apply() to PTransform.expand()

See https://issues.apache.org/jira/browse/BEAM-1125


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e62249a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e62249a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e62249a1

Branch: refs/heads/python-sdk
Commit: e62249a1f0a170f7e16926a3f0e6bc25d1422c22
Parents: d3c8874
Author: Ahmet Altay 
Authored: Thu Dec 15 14:27:08 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 16:51:50 2016 -0800

--
 sdks/python/README.md   |  2 +-
 .../examples/complete/autocomplete.py   |  2 +-
 .../examples/complete/estimate_pi.py|  2 +-
 .../apache_beam/examples/complete/tfidf.py  |  2 +-
 .../examples/complete/top_wikipedia_sessions.py |  6 ++---
 .../examples/cookbook/custom_ptransform.py  |  2 +-
 .../examples/cookbook/multiple_output_pardo.py  |  2 +-
 .../apache_beam/examples/snippets/snippets.py   | 16 ++---
 .../examples/snippets/snippets_test.py  |  2 +-
 .../apache_beam/examples/wordcount_debugging.py |  2 +-
 sdks/python/apache_beam/io/avroio.py|  4 ++--
 .../apache_beam/io/datastore/v1/datastoreio.py  |  4 ++--
 sdks/python/apache_beam/io/iobase.py|  6 ++---
 sdks/python/apache_beam/io/textio.py|  4 ++--
 sdks/python/apache_beam/pipeline_test.py|  4 ++--
 .../runners/dataflow/native_io/iobase.py|  2 +-
 .../apache_beam/runners/direct/direct_runner.py |  2 +-
 sdks/python/apache_beam/runners/runner.py   |  4 ++--
 sdks/python/apache_beam/transforms/combiners.py | 14 ++--
 .../apache_beam/transforms/combiners_test.py|  2 +-
 sdks/python/apache_beam/transforms/core.py  | 24 ++--
 .../python/apache_beam/transforms/ptransform.py | 10 
 .../apache_beam/transforms/ptransform_test.py   |  6 ++---
 .../python/apache_beam/transforms/sideinputs.py | 10 
 sdks/python/apache_beam/transforms/util.py  |  4 ++--
 .../transforms/write_ptransform_test.py |  2 +-
 .../typehints/typed_pipeline_test.py|  2 +-
 27 files changed, 71 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e62249a1/sdks/python/README.md
--
diff --git a/sdks/python/README.md b/sdks/python/README.md
index 820084d..5ea2a60 100644
--- a/sdks/python/README.md
+++ b/sdks/python/README.md
@@ -262,7 +262,7 @@ import re
 import apache_beam as beam
 p = beam.Pipeline('DirectPipelineRunner')
 class MyCountTransform(beam.PTransform):
-  def apply(self, pcoll):
+  def expand(self, pcoll):
 return (pcoll
 | 'one word' >> beam.Map(lambda word: (word, 1))
 # GroupByKey accepts a PCollection of (word, 1) elements and

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e62249a1/sdks/python/apache_beam/examples/complete/autocomplete.py
--
diff --git a/sdks/python/apache_beam/examples/complete/autocomplete.py 
b/sdks/python/apache_beam/examples/complete/autocomplete.py
index c3cd88f..eaa5ca2 100644
--- a/sdks/python/apache_beam/examples/complete/autocomplete.py
+++ b/sdks/python/apache_beam/examples/complete/autocomplete.py
@@ -60,7 +60,7 @@ class TopPerPrefix(beam.PTransform):
 super(TopPerPrefix, self).__init__()
 self._count = count
 
-  def apply(self, words):
+  def expand(self, words):
 """Compute the most common words for each possible prefixes.
 
 Args:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e62249a1/sdks/python/apache_beam/examples/complete/estimate_pi.py
--
diff --git a/sdks/python/apache_beam/examples/complete/estimate_pi.py 
b/sdks/python/apache_beam/examples/complete/estimate_pi.py
index 37c1aad..682c6d2 100644
--- a/sdks/python/apache_beam/examples/complete/estimate_pi.py
+++ b/sdks/python/apache_beam/examples/complete/estimate_pi.py
@@ -90,7 +90,7 @@ class JsonCoder(object):
 class EstimatePiTransform(beam.PTransform):
   """Runs 10M trials, and combine the results to estimate pi."""
 
-  def apply(self, pcoll):
+  def expand(self, pcoll):
 # A hundred work items of a hundred thousand tries each.
 return (pcoll
 | 'Initialize' >> beam.Create([10] * 
100).with_output_types(int)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e62249a1/sdks/python/apache_beam/examples/complete/tfidf.py

[2/2] incubator-beam git commit: Closes #1634

[robertwb]

Closes #1634


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e2652787
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e2652787
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e2652787

Branch: refs/heads/python-sdk
Commit: e2652787355d4c322138f55ae2c54494ec592e59
Parents: d3c8874 e62249a
Author: Robert Bradshaw 
Authored: Thu Dec 15 16:52:39 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 16:52:39 2016 -0800

--
 sdks/python/README.md   |  2 +-
 .../examples/complete/autocomplete.py   |  2 +-
 .../examples/complete/estimate_pi.py|  2 +-
 .../apache_beam/examples/complete/tfidf.py  |  2 +-
 .../examples/complete/top_wikipedia_sessions.py |  6 ++---
 .../examples/cookbook/custom_ptransform.py  |  2 +-
 .../examples/cookbook/multiple_output_pardo.py  |  2 +-
 .../apache_beam/examples/snippets/snippets.py   | 16 ++---
 .../examples/snippets/snippets_test.py  |  2 +-
 .../apache_beam/examples/wordcount_debugging.py |  2 +-
 sdks/python/apache_beam/io/avroio.py|  4 ++--
 .../apache_beam/io/datastore/v1/datastoreio.py  |  4 ++--
 sdks/python/apache_beam/io/iobase.py|  6 ++---
 sdks/python/apache_beam/io/textio.py|  4 ++--
 sdks/python/apache_beam/pipeline_test.py|  4 ++--
 .../runners/dataflow/native_io/iobase.py|  2 +-
 .../apache_beam/runners/direct/direct_runner.py |  2 +-
 sdks/python/apache_beam/runners/runner.py   |  4 ++--
 sdks/python/apache_beam/transforms/combiners.py | 14 ++--
 .../apache_beam/transforms/combiners_test.py|  2 +-
 sdks/python/apache_beam/transforms/core.py  | 24 ++--
 .../python/apache_beam/transforms/ptransform.py | 10 
 .../apache_beam/transforms/ptransform_test.py   |  6 ++---
 .../python/apache_beam/transforms/sideinputs.py | 10 
 sdks/python/apache_beam/transforms/util.py  |  4 ++--
 .../transforms/write_ptransform_test.py |  2 +-
 .../typehints/typed_pipeline_test.py|  2 +-
 27 files changed, 71 insertions(+), 71 deletions(-)
--

[jira] [Resolved] (BEAM-1153) GcsUtil needs to set timeout and retry explicitly in BatchRequest.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1153.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> GcsUtil needs to set timeout and retry explicitly in BatchRequest.
> --
>
> Key: BEAM-1153
> URL: https://issues.apache.org/jira/browse/BEAM-1153
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> Non-batch requests uses RetryHttpRequestInitializer, which set read timeout 
> as 80 seconds, and does more retries.
> Google Cloud auto generated Json library doesn't set HttpRequestInitializer 
> for batch requests.
> GcsUtil uses storageClient.batch(), and it is defined in here:
> https://github.com/vparfonov/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/services/AbstractGoogleClient.java#L256
> Without the HttpRequestInitializer, the default read timeout is 20 seconds.
> Possible fix is: https://github.com/apache/incubator-beam/pull/1608
> In additional, we can partially rollback 
> https://github.com/apache/incubator-beam/pull/1359 to keep using non-batch 
> API for fileSize() for single files. This will make sure existing code will 
> keep work as the same way.
> PR: https://github.com/apache/incubator-beam/pull/1611



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1637: [BEAM-1153] GcsUtil: use non-batch API fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1637


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1153] GcsUtil: use non-batch API for single file size requests.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating f753422a3 -> bce0a0d50


[BEAM-1153] GcsUtil: use non-batch API for single file size requests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/58601f8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/58601f8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/58601f8c

Branch: refs/heads/release-0.4.0-incubating
Commit: 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Parents: f753422
Author: Pei He 
Authored: Tue Dec 13 18:29:17 2016 -0800
Committer: Pei He 
Committed: Thu Dec 15 16:30:54 2016 -0800

--
 .../java/org/apache/beam/sdk/util/GcsUtil.java  | 29 -
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 65 +++-
 2 files changed, 92 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/58601f8c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
index 2edb1d6..dcdba46 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
@@ -267,7 +267,34 @@ public class GcsUtil {
* if the resource does not exist.
*/
   public long fileSize(GcsPath path) throws IOException {
-return fileSizes(ImmutableList.of(path)).get(0);
+return fileSize(
+path,
+BACKOFF_FACTORY.backoff(),
+Sleeper.DEFAULT);
+  }
+
+  /**
+   * Returns the file size from GCS or throws {@link FileNotFoundException}
+   * if the resource does not exist.
+   */
+  @VisibleForTesting
+  long fileSize(GcsPath path, BackOff backoff, Sleeper sleeper) throws 
IOException {
+Storage.Objects.Get getObject =
+storageClient.objects().get(path.getBucket(), path.getObject());
+try {
+  StorageObject object = ResilientOperation.retry(
+  ResilientOperation.getGoogleRequestCallable(getObject),
+  backoff,
+  RetryDeterminer.SOCKET_ERRORS,
+  IOException.class,
+  sleeper);
+  return object.getSize().longValue();
+} catch (Exception e) {
+  if (e instanceof IOException && 
errorExtractor.itemNotFound((IOException) e)) {
+throw new FileNotFoundException(path.toString());
+  }
+  throw new IOException("Unable to get file size", e);
+}
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/58601f8c/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
index c8ed402..6ca87f9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
@@ -57,6 +57,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import java.io.FileNotFoundException;
 import java.io.IOException;
+import java.math.BigInteger;
 import java.net.SocketTimeoutException;
 import java.nio.channels.SeekableByteChannel;
 import java.nio.file.AccessDeniedException;
@@ -320,7 +321,69 @@ public class GcsUtilTest {
   }
 
   @Test
-  public void testGetSizeBytesWhenFileNotFound() throws Exception {
+  public void testFileSizeNonBatch() throws Exception {
+GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
+GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
+
+Storage mockStorage = Mockito.mock(Storage.class);
+gcsUtil.setStorageClient(mockStorage);
+
+Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class);
+Storage.Objects.Get mockStorageGet = 
Mockito.mock(Storage.Objects.Get.class);
+
+when(mockStorage.objects()).thenReturn(mockStorageObjects);
+when(mockStorageObjects.get("testbucket", 
"testobject")).thenReturn(mockStorageGet);
+when(mockStorageGet.execute()).thenReturn(
+new StorageObject().setSize(BigInteger.valueOf(1000)));
+
+assertEquals(1000, gcsUtil.fileSize(GcsPath.fromComponents("testbucket", 
"testobject")));
+  }
+
+  @Test
+  public void testFileSizeWhenFileNotFoundNonBatch() throws Exception {
+MockLowLevelHttpResponse notFoundResponse = new MockLowLevelHttpResponse();
+notFoundResponse.setContent("");
+notFoundResponse.setStatusCode(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+
+MockHttpTransport mockTransport =
+   

[jira] [Commented] (BEAM-1153) GcsUtil needs to set timeout and retry explicitly in BatchRequest.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753013#comment-15753013
 ] 

ASF GitHub Bot commented on BEAM-1153:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1637


> GcsUtil needs to set timeout and retry explicitly in BatchRequest.
> --
>
> Key: BEAM-1153
> URL: https://issues.apache.org/jira/browse/BEAM-1153
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>Priority: Blocker
>
> Non-batch requests uses RetryHttpRequestInitializer, which set read timeout 
> as 80 seconds, and does more retries.
> Google Cloud auto generated Json library doesn't set HttpRequestInitializer 
> for batch requests.
> GcsUtil uses storageClient.batch(), and it is defined in here:
> https://github.com/vparfonov/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/services/AbstractGoogleClient.java#L256
> Without the HttpRequestInitializer, the default read timeout is 20 seconds.
> Possible fix is: https://github.com/apache/incubator-beam/pull/1608
> In additional, we can partially rollback 
> https://github.com/apache/incubator-beam/pull/1359 to keep using non-batch 
> API for fileSize() for single files. This will make sure existing code will 
> keep work as the same way.
> PR: https://github.com/apache/incubator-beam/pull/1611



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1637

[davor]

This closes #1637


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bce0a0d5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bce0a0d5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bce0a0d5

Branch: refs/heads/release-0.4.0-incubating
Commit: bce0a0d50bdb4e560790c8292f031f5757a9bb8c
Parents: f753422 58601f8
Author: Davor Bonaci 
Authored: Thu Dec 15 16:49:07 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 16:49:07 2016 -0800

--
 .../java/org/apache/beam/sdk/util/GcsUtil.java  | 29 -
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 65 +++-
 2 files changed, 92 insertions(+), 2 deletions(-)
--

[jira] [Updated] (BEAM-1084) Update apitools to version 0.5.6

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-1084:

Description: There are some fixes to JsonValue that should be included in 
beam (https://github.com/google/apitools/pull/136)   (was: There are some fixes 
to JsonValue that should be included in beam.)

> Update apitools to version 0.5.6
> 
>
> Key: BEAM-1084
> URL: https://issues.apache.org/jira/browse/BEAM-1084
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>
> There are some fixes to JsonValue that should be included in beam 
> (https://github.com/google/apitools/pull/136) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1617: [BEAM-964] json avro flag migration

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1617


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752986#comment-15752986
 ] 

ASF GitHub Bot commented on BEAM-964:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1617


> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1617

[robertwb]

Closes #1617


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d3c88748
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d3c88748
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d3c88748

Branch: refs/heads/python-sdk
Commit: d3c88748099fcccb27aef67c5c390d0bc67ebeb0
Parents: e383c77 0a558c7
Author: Robert Bradshaw 
Authored: Thu Dec 15 16:35:59 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 16:35:59 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Update the BQ export flat from Json to Avro

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e383c7715 -> d3c887480


Update the BQ export flat from Json to Avro


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a558c71
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a558c71
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a558c71

Branch: refs/heads/python-sdk
Commit: 0a558c7171d6e4452d88ecffd16a024a19cbfc42
Parents: e383c77
Author: Sourabh Bajaj 
Authored: Wed Dec 14 11:44:46 2016 -0800
Committer: Sourabh Bajaj 
Committed: Wed Dec 14 11:44:46 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a558c71/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 8b953b0..a3f7d94 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -520,7 +520,7 @@ class DataflowPipelineRunner(PipelineRunner):
 elif transform.source.format == 'text':
   step.add_property(PropertyNames.FILE_PATTERN, transform.source.path)
 elif transform.source.format == 'bigquery':
-  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_JSON')
+  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_AVRO')
   # TODO(silviuc): Add table validation if transform.source.validate.
   if transform.source.table_reference is not None:
 step.add_property(PropertyNames.BIGQUERY_DATASET,

[jira] [Commented] (BEAM-1153) GcsUtil needs to set timeout and retry explicitly in BatchRequest.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752972#comment-15752972
 ] 

ASF GitHub Bot commented on BEAM-1153:
--

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1637

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam release-0.4.0-incubating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1637


commit 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Author: Pei He 
Date:   2016-12-14T02:29:17Z

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.




> GcsUtil needs to set timeout and retry explicitly in BatchRequest.
> --
>
> Key: BEAM-1153
> URL: https://issues.apache.org/jira/browse/BEAM-1153
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>Priority: Blocker
>
> Non-batch requests uses RetryHttpRequestInitializer, which set read timeout 
> as 80 seconds, and does more retries.
> Google Cloud auto generated Json library doesn't set HttpRequestInitializer 
> for batch requests.
> GcsUtil uses storageClient.batch(), and it is defined in here:
> https://github.com/vparfonov/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/services/AbstractGoogleClient.java#L256
> Without the HttpRequestInitializer, the default read timeout is 20 seconds.
> Possible fix is: https://github.com/apache/incubator-beam/pull/1608
> In additional, we can partially rollback 
> https://github.com/apache/incubator-beam/pull/1359 to keep using non-batch 
> API for fileSize() for single files. This will make sure existing code will 
> keep work as the same way.
> PR: https://github.com/apache/incubator-beam/pull/1611



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1637: [BEAM-1153] GcsUtil: use non-batch API fo...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1637

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam release-0.4.0-incubating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1637


commit 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Author: Pei He 
Date:   2016-12-14T02:29:17Z

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1636: [BEAM-498] Moves OldDoFn to runners-core

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1636

[BEAM-498] Moves OldDoFn to runners-core

This will of course need the usual Dataflow worker surgery.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam move-old-do-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1636


commit 374a450a66baf43786e2a22afb8e8832d3146441
Author: Eugene Kirpichov 
Date:   2016-12-16T00:16:46Z

Moves OldDoFn to runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2115

[Apache Jenkins Server]

See 

--
[...truncated 5829 lines...]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.384 sec - in 
org.apache.beam.sdk.transforms.LatestTest
Running org.apache.beam.sdk.transforms.MapElementsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 sec - in 
org.apache.beam.sdk.transforms.MapElementsTest
Running org.apache.beam.sdk.transforms.WithTimestampsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.321 sec - in 
org.apache.beam.sdk.transforms.WithTimestampsTest
Running org.apache.beam.sdk.transforms.TopTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.643 sec - in 
org.apache.beam.sdk.transforms.TopTest
Running org.apache.beam.sdk.transforms.FlatMapElementsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec - in 
org.apache.beam.sdk.transforms.FlatMapElementsTest
Running org.apache.beam.sdk.transforms.ApproximateUniqueTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.873 sec - in 
org.apache.beam.sdk.transforms.ApproximateUniqueTest
Running org.apache.beam.sdk.transforms.ViewTest
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.125 sec - in 
org.apache.beam.sdk.transforms.ViewTest
Running org.apache.beam.sdk.transforms.CombineFnsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in 
org.apache.beam.sdk.transforms.CombineFnsTest
Running org.apache.beam.sdk.transforms.GroupByKeyTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.28 sec - in 
org.apache.beam.sdk.transforms.GroupByKeyTest
Running org.apache.beam.sdk.transforms.FilterTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.26 sec - in 
org.apache.beam.sdk.transforms.FilterTest
Running org.apache.beam.sdk.transforms.ParDoLifecycleTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.388 sec - in 
org.apache.beam.sdk.transforms.ParDoLifecycleTest
Running org.apache.beam.sdk.transforms.ApproximateQuantilesTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.282 sec - in 
org.apache.beam.sdk.transforms.ApproximateQuantilesTest
Running org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.285 sec - in 
org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
Running org.apache.beam.sdk.transforms.DoFnTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec - in 
org.apache.beam.sdk.transforms.DoFnTest
Running org.apache.beam.sdk.transforms.windowing.WindowTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec - in 
org.apache.beam.sdk.transforms.windowing.WindowTest
Running org.apache.beam.sdk.transforms.windowing.WindowingTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.332 sec - in 
org.apache.beam.sdk.transforms.windowing.WindowingTest
Running org.apache.beam.sdk.transforms.ValuesTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec - in 
org.apache.beam.sdk.transforms.ValuesTest
Running org.apache.beam.sdk.transforms.SplittableDoFnTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.103 sec - in 
org.apache.beam.sdk.transforms.SplittableDoFnTest
Running org.apache.beam.sdk.transforms.RegexTest
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.36 sec - in 
org.apache.beam.sdk.transforms.RegexTest
Running org.apache.beam.sdk.transforms.PartitionTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.316 sec - in 
org.apache.beam.sdk.transforms.PartitionTest
Running org.apache.beam.sdk.transforms.ParDoTest
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.836 sec - in 
org.apache.beam.sdk.transforms.ParDoTest
Running org.apache.beam.sdk.transforms.CreateTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.DistinctTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.153 sec - in 
org.apache.beam.sdk.transforms.DistinctTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.046 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.transforms.KvSwapTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec - in 
org.apache.beam.sdk.transforms.KvSwapTest
Running org.apache.beam.sdk.util.ReshuffleTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in 
org.apache.beam.sdk.util.ReshuffleTest
Running org.apache.beam.runners.core.UnboundedReadFromBoundedSourceTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.302 sec - in 
org.apache.beam.runners.core.UnboundedReadFromBoundedSourceTest

Results :

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752949#comment-15752949
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1636

[BEAM-498] Moves OldDoFn to runners-core

This will of course need the usual Dataflow worker surgery.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam move-old-do-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1636


commit 374a450a66baf43786e2a22afb8e8832d3146441
Author: Eugene Kirpichov 
Date:   2016-12-16T00:16:46Z

Moves OldDoFn to runners-core




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1635: Remove deprecated methods of InMemoryTime...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1635

Remove deprecated methods of InMemoryTimerInternals

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tweise just choosing reviewer at random
CC: @scwhittle for the deprecation which I think is now cleaned up

(no JIRA because it is just cleanup)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
remove-deprecated-TimerInternals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1635.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1635


commit e26f4075af6f2c990e23dc9f8fc8be2233652a9f
Author: Kenneth Knowles 
Date:   2016-12-16T00:02:23Z

Remove deprecated methods of InMemoryTimerInternals




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1625: Don't incorrectly log error in MetricsEnv...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1625


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1634: [BEAM-1125] Rename PTransform.apply() to ...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1634

[BEAM-1125] Rename PTransform.apply() to PTransform.expand()

Rename apply function to expand to match the recent change in the Java SDK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit b6a0974375e5c7414a52cd3ffd2a9e9fe8d1889f
Author: Ahmet Altay 
Date:   2016-12-15T22:27:08Z

Rename PTransform.apply() to PTransform.expand()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1125) Rename PTransform.apply to PTransform.expand

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752796#comment-15752796
 ] 

ASF GitHub Bot commented on BEAM-1125:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1634

[BEAM-1125] Rename PTransform.apply() to PTransform.expand()

Rename apply function to expand to match the recent change in the Java SDK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit b6a0974375e5c7414a52cd3ffd2a9e9fe8d1889f
Author: Ahmet Altay 
Date:   2016-12-15T22:27:08Z

Rename PTransform.apply() to PTransform.expand()




> Rename PTransform.apply to PTransform.expand
> 
>
> Key: BEAM-1125
> URL: https://issues.apache.org/jira/browse/BEAM-1125
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
>
> For context see:
> [BEAM-438] https://issues.apache.org/jira/browse/BEAM-438
> [PR #1538] https://github.com/apache/incubator-beam/pull/1538
> https://lists.apache.org/thread.html/b4d9bcfbfeaa5dbcd5b68fd2344cdffe45587ff88cb714638504e759@%3Cdev.beam.apache.org%3E
> This requires renaming the apply method, updating all custom PTransforms, and 
> runners where transform.apply is called. (Based on the Java PR, this could be 
> easily done with a refactoring tool.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1120) Prepare Dataflow runner for 0.4.0 release

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1120.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Prepare Dataflow runner for 0.4.0 release
> -
>
> Key: BEAM-1120
> URL: https://issues.apache.org/jira/browse/BEAM-1120
> Project: Beam
>  Issue Type: Improvement
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1166) Source.getDefaultOutputCoder() documentation should mention CannotProvideCoderException

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1166:
--
Component/s: sdk-java-core
 beam-model

> Source.getDefaultOutputCoder() documentation should mention 
> CannotProvideCoderException
> ---
>
> Key: BEAM-1166
> URL: https://issues.apache.org/jira/browse/BEAM-1166
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Knowing that you can throw CannotProviderCoderException is an important part 
> of implementing getDefaultOutputCoder
> The documentation for PTransform's getDefaultOutputCoder mentions this class, 
> and we should do this for the Source class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2111

[Apache Jenkins Server]

See 

--
Started by user davor
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752696#comment-15752696
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1565


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1565: [BEAM-498] Practically removes OldDoFn fr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1565


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[01/10] incubator-beam git commit: Converts all easy OldDoFns to DoFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3e1a62815 -> 5a3ace4a7


Converts all easy OldDoFns to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5f329ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5f329ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5f329ee

Branch: refs/heads/master
Commit: f5f329eee4e4a446dafe15b1c42a8f0972360fbc
Parents: 3e1a628
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:17:46 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:48:27 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java| 11 ++
 .../FlattenPCollectionTranslatorTest.java   | 15 +++-
 .../translation/GroupByKeyTranslatorTest.java   | 21 ---
 .../translation/ParDoBoundTranslatorTest.java   | 39 ++--
 .../translation/ReadUnboundTranslatorTest.java  | 15 +++-
 .../apache/beam/runners/flink/FlinkRunner.java  | 10 ++---
 .../beam/runners/flink/PipelineOptionsTest.java | 11 +++---
 .../flink/streaming/DoFnOperatorTest.java   | 19 +-
 .../flink/streaming/GroupByNullKeyTest.java | 18 -
 .../streaming/TopWikipediaSessionsITCase.java   | 10 ++---
 10 files changed, 75 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5f329ee/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index 899efa3..e5bde46 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -22,13 +22,11 @@ import com.datatorrent.api.Context.DAGContext;
 import com.datatorrent.api.DAG;
 import com.datatorrent.api.StreamingApplication;
 import com.google.common.base.Throwables;
-
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.concurrent.atomic.AtomicReference;
-
 import org.apache.apex.api.EmbeddedAppLauncher;
 import org.apache.apex.api.Launcher;
 import org.apache.apex.api.Launcher.AppHandle;
@@ -45,7 +43,6 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -245,10 +242,10 @@ public class ApexRunner extends 
PipelineRunner {
 }
   }
 
-  private static class WrapAsList extends OldDoFn {
-@Override
+  private static class WrapAsList extends DoFn {
+@ProcessElement
 public void processElement(ProcessContext c) {
-  c.output(Arrays.asList(c.element()));
+  c.output(Collections.singletonList(c.element()));
 }
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5f329ee/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
--
diff --git 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
index 6b62a58..f5abc34 100644
--- 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
+++ 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
@@ -19,12 +19,11 @@
 package org.apache.beam.runners.apex.translation;
 
 import com.google.common.collect.Sets;
-
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.Set;
-
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.ApexRunner;
 import org.apache.beam.runners.apex.ApexRunnerResult;
@@ -32,8 +31,8 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import 

[04/10] incubator-beam git commit: Removes code for wrapping DoFn as an OldDoFn

[kenn]

Removes code for wrapping DoFn as an OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a22de150
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a22de150
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a22de150

Branch: refs/heads/master
Commit: a22de15012c51e8b7e31143021f0a298e093bf51
Parents: e9e53c5
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:21:40 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../beam/sdk/transforms/DoFnAdapters.java   | 150 --
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 295 +--
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 +
 .../transforms/reflect/DoFnInvokersTest.java|  36 ---
 4 files changed, 11 insertions(+), 611 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a22de150/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index e15b08b..d1c40a6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -18,8 +18,6 @@
 package org.apache.beam.sdk.transforms;
 
 import java.io.IOException;
-import java.util.Collection;
-import javax.annotation.Nullable;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn.Context;
@@ -38,7 +36,6 @@ import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.TypeDescriptor;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 
@@ -53,18 +50,6 @@ public class DoFnAdapters {
   /** Should not be instantiated. */
   private DoFnAdapters() {}
 
-  /**
-   * If this is an {@link OldDoFn} produced via {@link #toOldDoFn}, returns 
the class of the
-   * original {@link DoFn}, otherwise returns {@code fn.getClass()}.
-   */
-  public static Class getDoFnClass(OldDoFn fn) {
-if (fn instanceof SimpleDoFnAdapter) {
-  return ((SimpleDoFnAdapter) fn).fn.getClass();
-} else {
-  return fn.getClass();
-}
-  }
-
   /** Creates an {@link OldDoFn} that delegates to the {@link DoFn}. */
   @SuppressWarnings({"unchecked", "rawtypes"})
   public static  OldDoFn 
toOldDoFn(DoFn fn) {
@@ -76,126 +61,6 @@ public class DoFnAdapters {
 }
   }
 
-  /** Creates a {@link OldDoFn.ProcessContext} from a {@link 
DoFn.ProcessContext}. */
-  public static  OldDoFn.ProcessContext 
adaptProcessContext(
-  OldDoFn fn,
-  final DoFn.ProcessContext c,
-  final DoFnInvoker.ArgumentProvider extra) {
-return fn.new ProcessContext() {
-  @Override
-  public InputT element() {
-return c.element();
-  }
-
-  @Override
-  public  T sideInput(PCollectionView view) {
-return c.sideInput(view);
-  }
-
-  @Override
-  public Instant timestamp() {
-return c.timestamp();
-  }
-
-  @Override
-  public BoundedWindow window() {
-return extra.window();
-  }
-
-  @Override
-  public PaneInfo pane() {
-return c.pane();
-  }
-
-  @Override
-  public WindowingInternals windowingInternals() {
-return extra.windowingInternals();
-  }
-
-  @Override
-  public PipelineOptions getPipelineOptions() {
-return c.getPipelineOptions();
-  }
-
-  @Override
-  public void output(OutputT output) {
-c.output(output);
-  }
-
-  @Override
-  public void outputWithTimestamp(OutputT output, Instant timestamp) {
-c.outputWithTimestamp(output, timestamp);
-  }
-
-  @Override
-  public  void sideOutput(TupleTag tag, T output) {
-c.sideOutput(tag, output);
-  }
-
-  @Override
-  public  void sideOutputWithTimestamp(TupleTag tag, T output, 
Instant timestamp) {
-c.sideOutputWithTimestamp(tag, output, timestamp);
-  }
-
-  @Override
-  protected  Aggregator 
createAggregatorInternal(
-  String name, CombineFn combiner) {
-return 

[02/10] incubator-beam git commit: Pushes uses of OldDoFn deeper inside Flink runner

[kenn]

Pushes uses of OldDoFn deeper inside Flink runner

In particular, various DoFnOperator's now take a regular DoFn
rather than an OldDoFn, and convert it to an OldDoFn internally.

This allows to remove uses of ParDo.getFn() returning OldDoFn.

The only case where the OldDoFn inside a DoFnOperator is actually an
OldDoFn rather than DoFn in disguise is now WindowDoFnOperator, which
overrides getDoFn to return an actual GABW OldDoFn.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8330bfa7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8330bfa7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8330bfa7

Branch: refs/heads/master
Commit: 8330bfa74cd72e51a29649745e87a4f1a6e5ffa1
Parents: af616d9
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:47:01 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:55:24 2016 -0800

--
 .../FlinkBatchTransformTranslators.java |  9 +---
 .../FlinkStreamingTransformTranslators.java |  8 ++--
 .../functions/FlinkDoFnFunction.java| 10 +++--
 .../functions/FlinkMultiOutputDoFnFunction.java | 10 +++--
 .../wrappers/streaming/DoFnOperator.java| 43 
 .../wrappers/streaming/WindowDoFnOperator.java  |  8 ++--
 .../beam/runners/flink/PipelineOptionsTest.java |  5 +--
 .../flink/streaming/DoFnOperatorTest.java   |  8 ++--
 8 files changed, 63 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8330bfa7/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 9ac907f..497b293 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -50,7 +50,6 @@ import org.apache.beam.sdk.transforms.CombineFnBase;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -523,8 +522,6 @@ class FlinkBatchTransformTranslators {
   DataSet inputDataSet =
   context.getInputDataSet(context.getInput(transform));
 
-  final OldDoFn oldDoFn = transform.getFn();
-
   TypeInformation typeInformation =
   context.getTypeInfo(context.getOutput(transform));
 
@@ -539,7 +536,7 @@ class FlinkBatchTransformTranslators {
 
   FlinkDoFnFunction doFnWrapper =
   new FlinkDoFnFunction<>(
-  oldDoFn,
+  doFn,
   context.getOutput(transform).getWindowingStrategy(),
   sideInputStrategies,
   context.getPipelineOptions());
@@ -570,8 +567,6 @@ class FlinkBatchTransformTranslators {
   DataSet inputDataSet =
   context.getInputDataSet(context.getInput(transform));
 
-  final OldDoFn oldDoFn = transform.getFn();
-
   Map outputs = 
context.getOutput(transform).getAll();
 
   Map outputMap = Maps.newHashMap();
@@ -618,7 +613,7 @@ class FlinkBatchTransformTranslators {
   @SuppressWarnings("unchecked")
   FlinkMultiOutputDoFnFunction doFnWrapper =
   new FlinkMultiOutputDoFnFunction(
-  oldDoFn,
+  doFn,
   windowingStrategy,
   sideInputStrategies,
   context.getPipelineOptions(),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8330bfa7/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index 042f8df..42ef630 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
+++ 

[08/10] incubator-beam git commit: Moves DoFnAdapters to runners-core

[kenn]

Moves DoFnAdapters to runners-core


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33ed3238
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33ed3238
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33ed3238

Branch: refs/heads/master
Commit: 33ed3238e2b3899cff061be3056c5cc29fc60a04
Parents: ca1dd7a
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:28:16 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:59:11 2016 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../apache/beam/runners/core/DoFnAdapters.java  | 344 +++
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 340 --
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   2 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |   2 +-
 14 files changed, 367 insertions(+), 354 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
index 33b9269..ef049e1 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
@@ -22,8 +22,8 @@ import java.util.Collections;
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
+import org.apache.beam.runners.core.DoFnAdapters;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnAdapters;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 48ac177..4af7ff0 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -413,7 +413,7 @@ public class ApexGroupByKeyOperator implements 
Operator {
 }
 
 @Override
-protected  Aggregator 
createAggregatorInternal(
+public  Aggregator 
createAggregatorInternal(
 String name, Combine.CombineFn combiner) {
   throw new UnsupportedOperationException();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index 08f062d..1e76949 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -37,6 +37,7 @@ import 
org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.NoOpStepContext;
 import 

[07/10] incubator-beam git commit: Removes ArgumentProvider.windowingInternals

[kenn]

Removes ArgumentProvider.windowingInternals


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3e8a038
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3e8a038
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3e8a038

Branch: refs/heads/master
Commit: f3e8a0383bf9cb3f9452e0364f7deba113cadff9
Parents: a22de15
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:23:15 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 57 
 .../beam/runners/core/SplittableParDo.java  |  7 ---
 .../beam/sdk/transforms/DoFnAdapters.java   | 14 -
 .../apache/beam/sdk/transforms/DoFnTester.java  |  7 ---
 .../sdk/transforms/reflect/DoFnInvoker.java | 20 ---
 .../transforms/reflect/DoFnInvokersTest.java|  6 ---
 6 files changed, 111 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3e8a038/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index af7f5ca..041cdde 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -52,13 +52,10 @@ import 
org.apache.beam.sdk.util.ExecutionContext.StepContext;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.Timer;
-import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.State;
-import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 import org.apache.beam.sdk.util.state.StateSpec;
@@ -420,11 +417,6 @@ public class SimpleDoFnRunner implements 
DoFnRunner windowingInternals() {
-  throw new UnsupportedOperationException("WindowingInternals are 
unsupported.");
-}
-
-@Override
 public  RestrictionTracker 
restrictionTracker() {
   throw new UnsupportedOperationException(
   "Cannot access RestrictionTracker outside of @ProcessElement 
method.");
@@ -634,54 +626,5 @@ public class SimpleDoFnRunner implements 
DoFnRunner windowingInternals() {
-  return new WindowingInternals() {
-@Override
-public Collection windows() {
-  return windowedValue.getWindows();
-}
-
-@Override
-public PaneInfo pane() {
-  return windowedValue.getPane();
-}
-
-@Override
-public TimerInternals timerInternals() {
-  return context.stepContext.timerInternals();
-}
-
-@Override
-public StateInternals stateInternals() {
-  return stepContext.stateInternals();
-}
-
-@Override
-public void outputWindowedValue(
-OutputT output,
-Instant timestamp,
-Collection windows,
-PaneInfo pane) {
-  throw new UnsupportedOperationException("A DoFn cannot output to a 
different window");
-}
-
-@Override
-public  void sideOutputWindowedValue(
-TupleTag tag,
-SideOutputT output,
-Instant timestamp,
-Collection windows,
-PaneInfo pane) {
-  throw new UnsupportedOperationException(
-  "A DoFn cannot side output to a different window");
-}
-
-@Override
-public  T sideInput(PCollectionView view, BoundedWindow 
sideInputWindow) {
-  return context.sideInput(view, sideInputWindow);
-}
-  };
-}
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3e8a038/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 

[06/10] incubator-beam git commit: Removes unused code from NoOpOldDoFn

[kenn]

Removes unused code from NoOpOldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ca1dd7a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ca1dd7a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ca1dd7a3

Branch: refs/heads/master
Commit: ca1dd7a3a4a01e2696a8131809c5798aef55d6a0
Parents: f3e8a03
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:24:23 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java | 72 
 1 file changed, 72 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ca1dd7a3/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
index 302b66a..504480b 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
@@ -19,10 +19,6 @@ package org.apache.beam.sdk.transforms;
 
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowingInternals;
-import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 import org.joda.time.Instant;
 
@@ -46,13 +42,6 @@ class NoOpOldDoFn extends OldDoFn {
   }
 
   /**
-   * Returns a new NoOp Process Context.
-   */
-  public OldDoFn.ProcessContext processContext() {
-return new NoOpDoFnProcessContext();
-  }
-
-  /**
* A {@link OldDoFn.Context} that does nothing and returns exclusively null.
*/
   private class NoOpDoFnContext extends OldDoFn.Context {
@@ -79,65 +68,4 @@ class NoOpOldDoFn extends OldDoFn {
   return null;
 }
   }
-
-  /**
-   * A {@link OldDoFn.ProcessContext} that does nothing and returns exclusively
-   * null.
-   */
-  private class NoOpDoFnProcessContext extends OldDoFn.ProcessContext {
-@Override
-public InputT element() {
-  return null;
-}
-
-@Override
-public  T sideInput(PCollectionView view) {
-  return null;
-}
-
-@Override
-public Instant timestamp() {
-  return null;
-}
-
-@Override
-public BoundedWindow window() {
-  return null;
-}
-
-@Override
-public PaneInfo pane() {
-  return null;
-}
-
-@Override
-public WindowingInternals windowingInternals() {
-  return null;
-}
-
-@Override
-public PipelineOptions getPipelineOptions() {
-  return null;
-}
-
-@Override
-public void output(OutputT output) {}
-
-@Override
-public void outputWithTimestamp(OutputT output, Instant timestamp) {}
-
-@Override
-public  void sideOutput(TupleTag tag, T output) {}
-
-@Override
-public  void sideOutputWithTimestamp(TupleTag tag, T output,
-Instant timestamp) {}
-
-@Override
-protected  Aggregator
-createAggregatorInternal(String name, CombineFn combiner) {
-  return null;
-}
-
-  }
 }

[09/10] incubator-beam git commit: Renames ParDo.getNewFn to getFn

[kenn]

Renames ParDo.getNewFn to getFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b502fc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b502fc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b502fc1

Branch: refs/heads/master
Commit: 6b502fc111af266c7b1a0e6f7d473c36f57281a2
Parents: 33ed323
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:29:41 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:59:11 2016 -0800

--
 .../translation/ParDoBoundMultiTranslator.java  |  2 +-
 .../apex/translation/ParDoBoundTranslator.java  |  2 +-
 .../beam/runners/core/SplittableParDo.java  |  4 ++--
 .../runners/direct/ParDoEvaluatorFactory.java   |  2 +-
 .../direct/ParDoMultiOverrideFactory.java   |  2 +-
 .../ParDoSingleViaMultiOverrideFactory.java |  4 ++--
 .../direct/StatefulParDoEvaluatorFactory.java   |  4 ++--
 .../FlinkBatchTransformTranslators.java |  4 ++--
 .../FlinkStreamingTransformTranslators.java | 12 +--
 .../dataflow/DataflowPipelineTranslator.java|  8 +++
 .../spark/translation/TransformTranslator.java  |  4 ++--
 .../streaming/StreamingTransformTranslator.java |  4 ++--
 .../beam/sdk/AggregatorPipelineExtractor.java   |  4 ++--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 22 ++--
 .../sdk/AggregatorPipelineExtractorTest.java| 12 +--
 15 files changed, 45 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
index 574ce8f..bff7652 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
@@ -53,7 +53,7 @@ class ParDoBoundMultiTranslator
 
   @Override
   public void translate(ParDo.BoundMulti transform, 
TranslationContext context) {
-DoFn doFn = transform.getNewFn();
+DoFn doFn = transform.getFn();
 DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
 
 if (signature.stateDeclarations().size() > 0) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
index de78628..3b6eb6e 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
@@ -40,7 +40,7 @@ class ParDoBoundTranslator
 
   @Override
   public void translate(ParDo.Bound transform, 
TranslationContext context) {
-DoFn doFn = transform.getNewFn();
+DoFn doFn = transform.getFn();
 DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
 
 if (signature.stateDeclarations().size() > 0) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 720db63..f8d12ec 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -104,7 +104,7 @@ public class SplittableParDo
 checkNotNull(parDo, "parDo must not be null");
 this.parDo = parDo;
 checkArgument(
-
DoFnSignatures.getSignature(parDo.getNewFn().getClass()).processElement().isSplittable(),
+
DoFnSignatures.getSignature(parDo.getFn().getClass()).processElement().isSplittable(),
 "fn must be a splittable DoFn");
   }
 
@@ -114,7 +114,7 @@ public class 

[03/10] incubator-beam git commit: Remove ParDo.of(OldDoFn) from Apex runner

[kenn]

Remove ParDo.of(OldDoFn) from Apex runner

The only such usage was of AssignWindowsDoFn. Now, instead, it is
instantiated directly using a new translator for Window.Bound.

This change also separates the overloads of ApexParDoOperator for old
and new DoFn, to make the OldDoFn overload easier to track and later
remove.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/af616d97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/af616d97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/af616d97

Branch: refs/heads/master
Commit: af616d9741b19d0a7705df6fe075be1509aa659f
Parents: f5f329e
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:31:42 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:55:24 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java| 46 +---
 .../translation/ApexPipelineTranslator.java |  2 +
 .../translation/ParDoBoundMultiTranslator.java  |  4 +-
 .../apex/translation/ParDoBoundTranslator.java  |  4 +-
 .../apex/translation/WindowBoundTranslator.java | 78 
 .../operators/ApexParDoOperator.java| 25 ++-
 .../translation/ParDoBoundTranslatorTest.java   |  3 +-
 .../apache/beam/runners/core/AssignWindows.java | 46 
 .../spark/translation/SparkAssignWindowFn.java  |  2 +-
 9 files changed, 108 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af616d97/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index e5bde46..f12ebef 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -32,7 +32,6 @@ import org.apache.apex.api.Launcher;
 import org.apache.apex.api.Launcher.AppHandle;
 import org.apache.apex.api.Launcher.LaunchMode;
 import org.apache.beam.runners.apex.translation.ApexPipelineTranslator;
-import org.apache.beam.runners.core.AssignWindows;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderRegistry;
@@ -46,9 +45,6 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.PCollectionViews;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollection;
@@ -90,10 +86,7 @@ public class ApexRunner extends 
PipelineRunner {
   public  OutputT apply(
   PTransform transform, InputT input) {
 
-if (Window.Bound.class.equals(transform.getClass())) {
-  return (OutputT) ((PCollection) input).apply(
-  new AssignWindowsAndSetStrategy((Window.Bound) transform));
-} else if (Create.Values.class.equals(transform.getClass())) {
+if (Create.Values.class.equals(transform.getClass())) {
   return (OutputT) PCollection
   .createPrimitiveOutputInternal(
   input.getPipeline(),
@@ -162,43 +155,6 @@ public class ApexRunner extends 
PipelineRunner {
 
   }
 
-  /**
-   * copied from DirectPipelineRunner.
-   * used to replace Window.Bound till equivalent function is added in Apex
-   */
-  private static class AssignWindowsAndSetStrategy
-  extends PTransform {
-
-private final Window.Bound wrapped;
-
-public AssignWindowsAndSetStrategy(Window.Bound wrapped) {
-  this.wrapped = wrapped;
-}
-
-@Override
-public PCollection expand(PCollection input) {
-  WindowingStrategy outputStrategy =
-  wrapped.getOutputStrategyInternal(input.getWindowingStrategy());
-
-  WindowFn windowFn =
-  (WindowFn) outputStrategy.getWindowFn();
-
-  // If the Window.Bound transform only changed parts other than the 
WindowFn, then
-  // we skip AssignWindows even though it should be harmless in a perfect 
world.
-  // The world is not perfect, and a GBK may have set it to InvalidWindows 
to forcibly
-  // crash if another GBK is performed without explicitly setting the 
WindowFn. So we skip
-  // AssignWindows in this case.
-  if (wrapped.getWindowFn() == null) {
-return 

[05/10] incubator-beam git commit: Removes OldDoFn from ParDo

[kenn]

Removes OldDoFn from ParDo


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e9e53c5d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e9e53c5d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e9e53c5d

Branch: refs/heads/master
Commit: e9e53c5d037561aa4dcacfcde69d76a03f3a1571
Parents: 8330bfa
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:13:43 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 167 +++
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 --
 2 files changed, 55 insertions(+), 237 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e9e53c5d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 167f5fa..d2149c0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -27,7 +27,6 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.transforms.OldDoFn.RequiresWindowAccess;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
@@ -530,23 +529,6 @@ public class ParDo {
 return new Unbound().of(fn, displayDataForFn(fn));
   }
 
-  /**
-   * Creates a {@link ParDo} {@link PTransform} that will invoke the
-   * given {@link OldDoFn} function.
-   *
-   * The resulting {@link PTransform PTransform's} types have been bound, 
with the
-   * input being a {@code PCollection} and the output a
-   * {@code PCollection}, inferred from the types of the argument
-   * {@code OldDoFn}. It is ready to be applied, or further
-   * properties can be set on it first.
-   *
-   * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
-   */
-  @Deprecated
-  public static  Bound of(OldDoFn fn) {
-return new Unbound().of(fn, displayDataForFn(fn));
-  }
-
   private static  DisplayData.ItemSpec> 
displayDataForFn(T fn) {
 return DisplayData.item("fn", fn.getClass()).withLabel("Transform 
Function");
   }
@@ -557,12 +539,7 @@ public class ParDo {
* the {@link PCollection}.
*/
   private static  void validateWindowType(
-  PCollection input, Serializable fn) {
-// No validation for OldDoFn
-if (!(fn instanceof DoFn)) {
-  return;
-}
-
+  PCollection input, DoFn fn) {
 DoFnSignature signature = DoFnSignatures.getSignature((Class) 
fn.getClass());
 
 TypeDescriptor actualWindowT =
@@ -609,10 +586,6 @@ public class ParDo {
 }
   }
 
-  private static  OldDoFn adapt(DoFn fn) {
-return DoFnAdapters.toOldDoFn(fn);
-  }
-
   /**
* An incomplete {@link ParDo} transform, with unbound input/output types.
*
@@ -688,24 +661,9 @@ public class ParDo {
   return new UnboundMulti<>(name, sideInputs, mainOutputTag, 
sideOutputTags);
 }
 
-/**
- * Returns a new {@link ParDo} {@link PTransform} that's like this
- * transform but that will invoke the given {@link OldDoFn}
- * function, and that has its input and output types bound. Does
- * not modify this transform. The resulting {@link PTransform} is
- * sufficiently specified to be applied, but more properties can
- * still be specified.
- *
- * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
- */
-@Deprecated
-public  Bound of(OldDoFn oldFn) {
-  return of(oldFn, displayDataForFn(oldFn));
-}
-
 private  Bound of(
-Serializable originalFn, DisplayData.ItemSpec> 
fnDisplayData) {
-  return new Bound<>(name, originalFn, sideInputs, fnDisplayData);
+DoFn doFn, DisplayData.ItemSpec> 
fnDisplayData) {
+  return new Bound<>(name, doFn, sideInputs, fnDisplayData);
 }
   }
 
@@ -725,12 +683,12 @@ public class ParDo {
   extends PTransform {
 // Inherits name.
 private final List sideInputs;
-private 

[10/10] incubator-beam git commit: This closes #1565

[kenn]

This closes #1565

  Renames ParDo.getNewFn to getFn
  Moves DoFnAdapters to runners-core
  Removes unused code from NoOpOldDoFn
  Removes ArgumentProvider.windowingInternals
  Removes code for wrapping DoFn as an OldDoFn
  Removes OldDoFn from ParDo
  Pushes uses of OldDoFn deeper inside Flink runner
  Remove ParDo.of(OldDoFn) from Apex runner
  Converts all easy OldDoFns to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5a3ace4a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5a3ace4a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5a3ace4a

Branch: refs/heads/master
Commit: 5a3ace4a7c786938e5286e921f3afcd23b26de26
Parents: 3e1a628 6b502fc
Author: Kenneth Knowles 
Authored: Thu Dec 15 14:04:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 14:04:03 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java|  57 +--
 .../translation/ApexPipelineTranslator.java |   2 +
 .../translation/ParDoBoundMultiTranslator.java  |   6 +-
 .../apex/translation/ParDoBoundTranslator.java  |   6 +-
 .../apex/translation/WindowBoundTranslator.java |  78 +++
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|  25 +-
 .../FlattenPCollectionTranslatorTest.java   |  15 +-
 .../translation/GroupByKeyTranslatorTest.java   |  21 +-
 .../translation/ParDoBoundTranslatorTest.java   |  38 +-
 .../translation/ReadUnboundTranslatorTest.java  |  15 +-
 .../apache/beam/runners/core/AssignWindows.java |  46 --
 .../apache/beam/runners/core/DoFnAdapters.java  | 344 +
 .../beam/runners/core/SimpleDoFnRunner.java |  57 ---
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../beam/runners/core/SplittableParDo.java  |  11 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   2 +-
 .../direct/ParDoMultiOverrideFactory.java   |   2 +-
 .../ParDoSingleViaMultiOverrideFactory.java |   4 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   4 +-
 .../apache/beam/runners/flink/FlinkRunner.java  |  10 +-
 .../FlinkBatchTransformTranslators.java |  13 +-
 .../FlinkStreamingTransformTranslators.java |   4 +-
 .../functions/FlinkDoFnFunction.java|  10 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  10 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|  43 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   8 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   6 +-
 .../flink/streaming/DoFnOperatorTest.java   |  13 +-
 .../flink/streaming/GroupByNullKeyTest.java |  18 +-
 .../streaming/TopWikipediaSessionsITCase.java   |  10 +-
 .../dataflow/DataflowPipelineTranslator.java|   8 +-
 .../spark/translation/SparkAssignWindowFn.java  |   2 +-
 .../spark/translation/TransformTranslator.java  |   4 +-
 .../streaming/StreamingTransformTranslator.java |   4 +-
 .../beam/sdk/AggregatorPipelineExtractor.java   |   4 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 504 ---
 .../apache/beam/sdk/transforms/DoFnTester.java  |   7 -
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 297 +--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 173 +--
 .../sdk/transforms/reflect/DoFnInvoker.java |  20 -
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 +-
 .../sdk/AggregatorPipelineExtractorTest.java|  12 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |  74 +--
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++---
 .../transforms/reflect/DoFnInvokersTest.java|  42 --
 49 files changed, 689 insertions(+), 1629 deletions(-)
--

[jira] [Commented] (BEAM-450) Modules are shaded to the same path

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752612#comment-15752612
 ] 

ASF GitHub Bot commented on BEAM-450:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1633

[BEAM-450] Shade modules to separate paths

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam shading-package-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit 0bed1b3c494ec23f3ca18389d7b3bce15e0bd363
Author: Dan Halperin 
Date:   2016-12-15T21:50:39Z

[BEAM-450] Shade modules to separate paths




> Modules are shaded to the same path
> ---
>
> Key: BEAM-450
> URL: https://issues.apache.org/jira/browse/BEAM-450
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>  Labels: newbie, starter
>
> Right now multiple modules are using the same repackaged path. We should be 
> using per-artifact paths so that they don't conflict.
> One proposal was simply to adopt 
> {{${project.groupId}.${project.artifactId}.repackaged}} as the shading 
> location. If it works.
> This is a good starter issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1166) Source.getDefaultOutputCoder() documentation should mention CannotProvideCoderException

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1166:
--

 Summary: Source.getDefaultOutputCoder() documentation should 
mention CannotProvideCoderException
 Key: BEAM-1166
 URL: https://issues.apache.org/jira/browse/BEAM-1166
 Project: Beam
  Issue Type: Bug
Reporter: Stephen Sisk
Assignee: Stephen Sisk


Knowing that you can throw CannotProviderCoderException is an important part of 
implementing getDefaultOutputCoder

The documentation for PTransform's getDefaultOutputCoder mentions this class, 
and we should do this for the Source class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.0-incubating-RC2 [created] 76246ac4f

incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC2

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating a29affd05 -> f753422a3


[maven-release-plugin] rollback changes from release preparation of 
v0.4.0-incubating-RC2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f753422a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f753422a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f753422a

Branch: refs/heads/release-0.4.0-incubating
Commit: f753422a36ec6cbcf56dd6f69e1c7ba29e53a350
Parents: a29affd
Author: Davor Bonaci 
Authored: Thu Dec 15 13:22:04 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 13:22:04 2016 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 7333c10..e541edb 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 59d8f4d..8591955 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 65c6f1b..eb42861 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0d3db9d..e860c7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.4.0-incubating
+  0.4.0-incubating-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-v0.4.0-incubating-RC2
+release-0.4.0-incubating
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/runners/apex/pom.xml

[jira] [Created] (BEAM-1165) Unexpected file created when checking dependencies on clean repo

[JIRA]

Ismaël Mejía created BEAM-1165:
--

 Summary: Unexpected file created when checking dependencies on 
clean repo
 Key: BEAM-1165
 URL: https://issues.apache.org/jira/browse/BEAM-1165
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.5.0-incubating
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía
Priority: Minor


I just found a weird behavior when I was checking for the latest release,
nothing breaking, but when I start with a clean repo clone and I do:

mvn dependency:tree

It creates a new file runners/flink/examples/wordcounts.txt with the
dependencies.

This error happens because maven-dependency-plugin asumes the property output
used by the flink tests as the export file for the command.

Ref.
https://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html#output




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Apex #5

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1163) Add signature keys to the release guide vote template

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752444#comment-15752444
 ] 

ASF GitHub Bot commented on BEAM-1163:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam-site/pull/111

[BEAM-1163] Add signature keys to the vote template (release guide)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam-site BEAM-1163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/111.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #111


commit 4461b271569ae900372a7b137b66b64480968684
Author: Ismaël Mejía 
Date:   2016-12-15T20:32:09Z

[BEAM-1163] Add signature keys to the vote template (release guide)




> Add signature keys to the release guide vote template
> -
>
> Key: BEAM-1163
> URL: https://issues.apache.org/jira/browse/BEAM-1163
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Trivial
>
> A small improvement, the idea is to add just the fingerprint of the person 
> who signed the release in the template (for validation purposes:
> The release artifacts are signed with the key with fingerprint XXX
> https://dist.apache.org/repos/dist/release/incubator/beam/KEYS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #111: [BEAM-1163] Add signature keys to the...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam-site/pull/111

[BEAM-1163] Add signature keys to the vote template (release guide)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam-site BEAM-1163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/111.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #111


commit 4461b271569ae900372a7b137b66b64480968684
Author: Ismaël Mejía 
Date:   2016-12-15T20:32:09Z

[BEAM-1163] Add signature keys to the vote template (release guide)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752428#comment-15752428
 ] 

Daniel Halperin commented on BEAM-1023:
---

https://github.com/apache/incubator-beam/pull/1400

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752424#comment-15752424
 ] 

Daniel Halperin commented on BEAM-1023:
---

This was implemented by PR #1400, which was accidentally tagged with BEAM-1022 
instead of this issue.

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1023.
---
   Resolution: Fixed
Fix Version/s: 0.5.0-incubating

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Apex #4

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1630


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1022) TableNamespace should not use Java object equality when comparing windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752318#comment-15752318
 ] 

ASF GitHub Bot commented on BEAM-1022:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1400


> TableNamespace should not use Java object equality when comparing windows
> -
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.3.0-incubating
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-1022] Add testing coverage for BigQuery streaming writes

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3c4b6930e -> 3e1a62815


[BEAM-1022] Add testing coverage for BigQuery streaming writes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/51900830
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/51900830
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/51900830

Branch: refs/heads/master
Commit: 519008303f9cefd3f8f4a8a7a98a9a79717f57ff
Parents: 3c4b693
Author: Reuven Lax 
Authored: Thu Nov 17 10:57:41 2016 -0800
Committer: Dan Halperin 
Committed: Thu Dec 15 11:45:45 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  48 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 456 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 655 insertions(+), 628 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51900830/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 0be8567..28049ed 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -24,7 +24,6 @@ import static com.google.common.base.Preconditions.checkState;
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.api.client.json.JsonFactory;
-import com.google.api.services.bigquery.Bigquery;
 import com.google.api.services.bigquery.model.Job;
 import com.google.api.services.bigquery.model.JobConfigurationExtract;
 import com.google.api.services.bigquery.model.JobConfigurationLoad;
@@ -33,6 +32,7 @@ import 
com.google.api.services.bigquery.model.JobConfigurationTableCopy;
 import com.google.api.services.bigquery.model.JobReference;
 import com.google.api.services.bigquery.model.JobStatistics;
 import com.google.api.services.bigquery.model.JobStatus;
+import com.google.api.services.bigquery.model.Table;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
 import com.google.api.services.bigquery.model.TableSchema;
@@ -1796,8 +1796,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withCreateDisposition(CreateDisposition createDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -1806,8 +1806,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withWriteDisposition(WriteDisposition writeDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -2136,7 +2136,8 @@ public class BigQueryIO {
   /** Returns the table reference, or {@code null}. */
   @Nullable
   public ValueProvider getTable() {
-return NestedValueProvider.of(jsonTableRef, new 
JsonTableRefToTableRef());
+return jsonTableRef == null ? null :
+NestedValueProvider.of(jsonTableRef, new JsonTableRefToTableRef());
   }
 
   /** Returns {@code true} if table validation is enabled. */
@@ -2550,6 +2551,13 @@ public class BigQueryIO {
 }
   }
 
+  /**
+   * Clear the cached map of created tables. Used for testing.
+   */
+  @VisibleForTesting
+  static void clearCreatedTables() {
+StreamingWriteFn.clearCreatedTables();
+  }
   /
 
   /**
@@ -2585,6 +2593,15 @@ public class BigQueryIO {
   this.bqServices = 

[GitHub] incubator-beam pull request #1400: [BEAM-1022] Add test coverage for BigQuer...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1400


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1400

[dhalperi]

Closes #1400


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3e1a6281
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3e1a6281
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3e1a6281

Branch: refs/heads/master
Commit: 3e1a62815ca467951647788d59c00921bd02803a
Parents: 3c4b693 5190083
Author: Dan Halperin 
Authored: Thu Dec 15 11:46:03 2016 -0800
Committer: Dan Halperin 
Committed: Thu Dec 15 11:46:03 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  48 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 456 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 655 insertions(+), 628 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-1154] Get side input from proper window in ReduceFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating c09fbd43b -> 3ca8d2bfc


[BEAM-1154] Get side input from proper window in ReduceFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d57146f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d57146f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d57146f4

Branch: refs/heads/release-0.4.0-incubating
Commit: d57146f47297af2ad0759216a16898b43e0fa0af
Parents: c09fbd4
Author: Eugene Kirpichov 
Authored: Wed Dec 14 14:29:30 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:41:55 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d57146f4/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index c5bda9b..c71897d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -28,7 +28,6 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.ActiveWindowSet;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.TimeDomain;
@@ -98,11 +97,7 @@ class ReduceFnContextFactory {
 activeWindows,
 windowingStrategy.getWindowFn().windowCoder(),
 stateInternals,
-stateContextFromComponents(
-options,
-sideInputReader,
-window,
-windowingStrategy.getWindowFn()),
+stateContextFromComponents(options, sideInputReader, window),
 style);
   }
 
@@ -512,8 +507,7 @@ class ReduceFnContextFactory {
   private static  StateContext 
stateContextFromComponents(
   @Nullable final PipelineOptions options,
   final SideInputReader sideInputReader,
-  final W mainInputWindow,
-  final WindowFn windowFn) {
+  final W mainInputWindow) {
 if (options == null) {
   return StateContexts.nullContext();
 } else {
@@ -526,7 +520,11 @@ class ReduceFnContextFactory {
 
 @Override
 public  T sideInput(PCollectionView view) {
-  return sideInputReader.get(view, 
windowFn.getSideInputWindow(mainInputWindow));
+  return sideInputReader.get(
+  view,
+  view.getWindowingStrategyInternal()
+  .getWindowFn()
+  .getSideInputWindow(mainInputWindow));
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d57146f4/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index ba57567..4abfc9a 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -17,13 +17,14 @@
  */
 package org.apache.beam.runners.core;
 
-import static com.google.common.base.Preconditions.checkArgument;
 import static 
org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue;
 import static org.apache.beam.runners.core.WindowMatchers.isWindowedValue;
+import static org.hamcrest.Matchers.anyOf;
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.emptyIterable;
 import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertThat;
@@ -36,7 +37,6 @@ import 

[2/2] incubator-beam git commit: This closes #1630

[kenn]

This closes #1630


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3ca8d2bf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3ca8d2bf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3ca8d2bf

Branch: refs/heads/release-0.4.0-incubating
Commit: 3ca8d2bfc74199ac056a9ab94d6031ef97d746f9
Parents: c09fbd4 d57146f
Author: Kenneth Knowles 
Authored: Thu Dec 15 11:41:56 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:41:56 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--

[jira] [Commented] (BEAM-1154) ReduceFnRunner fetches side input from the wrong window

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752305#comment-15752305
 ] 

ASF GitHub Bot commented on BEAM-1154:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1620


> ReduceFnRunner fetches side input from the wrong window
> ---
>
> Key: BEAM-1154
> URL: https://issues.apache.org/jira/browse/BEAM-1154
> Project: Beam
>  Issue Type: Bug
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> This 
> https://github.com/apache/incubator-beam/blame/master/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java#L529
>  is incorrect - I broke it in 
> https://github.com/apache/incubator-beam/commit/90a0d0e13fa0332df805b79b1dc64860d9590217#diff-16edced77586e39a5f31907f4ced51b5R530
> It uses windowing strategy of the main input to do .getSideInputWindow() 
> instead of windowing strategy of the side input.
> The fix is very simple; trying to come up with a test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1620: [BEAM-1154] Get side input from proper wi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1620


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1620

[kenn]

This closes #1620


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3c4b6930
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3c4b6930
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3c4b6930

Branch: refs/heads/master
Commit: 3c4b6930e1e934ab41052796524ef68887cd7915
Parents: b38c9e9 de109d5
Author: Kenneth Knowles 
Authored: Thu Dec 15 11:40:59 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:40:59 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--

[GitHub] incubator-beam pull request #1631: Change counter name in TestDataflowRunner

[joshualitt]

GitHub user joshualitt opened a pull request:

https://github.com/apache/incubator-beam/pull/1631

Change counter name in TestDataflowRunner

This small change updates the name of the windmill watermark counter in 
TestDataflowRunner



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joshualitt/incubator-beam update_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1631.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1631


commit 2691a5878b00f2b0c313a4fb6d8d2c69d132effe
Author: Joshua Litt 
Date:   2016-12-15T19:07:27Z

Change counter name in TestDataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1625

[tgroh]

This closes #1625


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c09fbd43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c09fbd43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c09fbd43

Branch: refs/heads/release-0.4.0-incubating
Commit: c09fbd43b0cdaa57cffc1925c313dcf91390adc7
Parents: 5dcffe0 e864ac2
Author: Thomas Groh 
Authored: Thu Dec 15 11:12:01 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 15 11:12:01 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Don't incorrectly log error in MetricsEnvironment

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 5dcffe066 -> c09fbd43b


Don't incorrectly log error in MetricsEnvironment

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e864ac21
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e864ac21
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e864ac21

Branch: refs/heads/release-0.4.0-incubating
Commit: e864ac2141e31a930b85264506d06579ba6811de
Parents: 5dcffe0
Author: bchambers 
Authored: Wed Dec 14 11:23:39 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 15 11:12:00 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e864ac21/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
index 7c06cbf..5d7cb0b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
@@ -56,7 +56,7 @@ public class MetricsEnvironment {
*/
   @Nullable
   public static MetricsContainer setCurrentContainer(@Nullable 
MetricsContainer container) {
-MetricsContainer previous = getCurrentContainer();
+MetricsContainer previous = CONTAINER_FOR_THREAD.get();
 if (container == null) {
   CONTAINER_FOR_THREAD.remove();
 } else {

[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1630

Cherry pick #1620 to release-0.4.0-incubating

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The tests on #1620 have not come back yet, so please wait to merge until 
that is confirmed and merged on `master`. Just starting this PR to have 
something to reference.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
cherry-pick-CombineFnWithSideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1630.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1630


commit 7294f227b7b9ff660bcf8dbbb196662820438e36
Author: Eugene Kirpichov 
Date:   2016-12-14T22:29:30Z

[BEAM-1154] Get side input from proper window in ReduceFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1164) Allow a DoFn to opt in to mutating it's input

[Frances Perry (JIRA)]

Frances Perry created BEAM-1164:
---

 Summary: Allow a DoFn to opt in to mutating it's input
 Key: BEAM-1164
 URL: https://issues.apache.org/jira/browse/BEAM-1164
 Project: Beam
  Issue Type: Bug
  Components: beam-model
Reporter: Frances Perry
Priority: Minor


Runners generally can't tell if a DoFn is mutating inputs, but assuming so by 
default leads to significant performance implications from unnecessary copying 
(around sibling fusion, etc). So instead the model prevents mutating inputs, 
and the Direct Runner validates this behavior. (See: 
http://beam.incubator.apache.org/contribute/design-principles/#make-efficient-things-easy-rather-than-make-easy-things-efficient)
 

However, if users are processing a small number of large records by making 
incremental changes (for example, genomics use cases), the cost of immutability 
requirement can be very large. As a workaround, users sometimes do suboptimal 
things (fusing ParDos by hand) or undefined things when they expect the 
immutability requirement is unnecessarily strict (adding no-op coders in places 
they hope the runner won't be materializing things, mutating things anyway when 
they don't expect sibling fusion to happen, etc).

We should consider adding a signal (MutatingDoFn?) that users explicitly opt in 
to to say their code may mutate inputs. The runner can then use this assumption 
to either prevent optimizations that would break in the face of this or insert 
additional copies as needed to allow optimizations to preserve semantics.

See this related user@ discussion:
https://lists.apache.org/thread.html/f39689f54147117f3fc54c498eff1a20fa73f1be5b5cad5b6f816fd3@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1629

[davor]

This closes #1629


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5dcffe06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5dcffe06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5dcffe06

Branch: refs/heads/release-0.4.0-incubating
Commit: 5dcffe066944e8cc302e716337770551efa48aed
Parents: 7e234aa 21507b8
Author: Davor Bonaci 
Authored: Thu Dec 15 10:50:08 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 10:50:08 2016 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #1629: Set Dataflow worker version to 0.4.0-incu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1629


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Set Dataflow worker version to 0.4.0-incubating

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 7e234aa79 -> 5dcffe066


Set Dataflow worker version to 0.4.0-incubating


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/21507b8d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/21507b8d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/21507b8d

Branch: refs/heads/release-0.4.0-incubating
Commit: 21507b8da0bb74cc8d844ede985648ac4508df2b
Parents: 7e234aa
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:17:49 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:17:49 2016 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/21507b8d/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
index 1eae8cb..abfea0c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
+++ 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
@@ -18,6 +18,6 @@
 
 environment.major.version=6
 
-worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161205
+worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-0.4.0-incubating
 
-worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161205
+worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-0.4.0-incubating

[GitHub] incubator-beam pull request #1629: Set Dataflow worker version to 0.4.0-incu...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1629

Set Dataflow worker version to 0.4.0-incubating

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
bump-0.4.0-incubating-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1629.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1629


commit 21507b8da0bb74cc8d844ede985648ac4508df2b
Author: Kenneth Knowles 
Date:   2016-12-15T18:17:49Z

Set Dataflow worker version to 0.4.0-incubating




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1628


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1628

Bump 0.4.0 incubating worker

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
bump-0.4.0-incubating-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1628.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1628


commit 2c1d4eb4d878d1a8b8adf68b10eb8744cf524b5c
Author: Davor Bonaci 
Date:   2016-12-13T21:48:26Z

Update Maven Archetype versions to pin them to 0.4.0-incubating to be 
released

commit 9f3b0635971a45f8baf8fa68ba5ec2f8c47cec6b
Author: Davor Bonaci 
Date:   2016-12-13T22:18:32Z

Update Maven Archetype version; fixup for Java8.

commit 8d89bfce251d27c7fc6ff522a2cfeb676ec2c633
Author: Eugene Kirpichov 
Date:   2016-12-13T22:35:33Z

[BEAM-1149] Explode windows when fn uses side inputs

commit b2780881a6d735420a2d29c3270b781ea8a3b74f
Author: Kenneth Knowles 
Date:   2016-12-13T23:47:41Z

This closes #1602

commit f78d96069b428356609f219357cbf0702ec56c26
Author: Kenneth Knowles 
Date:   2016-12-14T00:09:57Z

Re-exclude UsesStatefulParDo tests for Dataflow

commit c1af44fa27633fd2a9592a13579415f6b974cfe6
Author: Kenneth Knowles 
Date:   2016-12-14T00:36:42Z

Revert "Allow stateful DoFn in DataflowRunner"

This reverts commit 42bb15d2df28b99b6788010450f41f2932095771.

The Dataflow service has introduced a bug that was masked by various
test disabling.

commit 10bb4767a1f989a1a75778828c07d9c72c450495
Author: Kenneth Knowles 
Date:   2016-12-14T01:08:42Z

This closes #1607

commit 122f05131c803ecaa082dfbc9ad6876b0ca467b8
Author: Kenneth Knowles 
Date:   2016-12-14T19:26:27Z

SimpleDoFnRunner observes window if SideInputReader is nonempty

commit d9f24b86c644ea85fd197eaab4c2d16b20a70d5f
Author: Kenneth Knowles 
Date:   2016-12-14T21:12:43Z

Fix NPE in StatefulParDoEvaluatorFactoryTest mocking

commit ea0d7f5c26fd97e8528d0614cdce8215ea05557c
Author: Kenneth Knowles 
Date:   2016-12-14T22:33:04Z

This closes #1619

commit af3989b086ed2e1bf0a08358ce6b7620c334cf49
Author: Davor Bonaci 
Date:   2016-12-15T02:58:57Z

[maven-release-plugin] prepare release v0.4.0-incubating-RC1

commit 7e234aa7938606d1a01064a6fdccf01bb90d4f2d
Author: Davor Bonaci 
Date:   2016-12-15T02:59:29Z

[maven-release-plugin] rollback changes from release preparation of 
v0.4.0-incubating-RC1

commit 21507b8da0bb74cc8d844ede985648ac4508df2b
Author: Kenneth Knowles 
Date:   2016-12-15T18:17:49Z

Set Dataflow worker version to 0.4.0-incubating




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1626: Fix typo in mvn command args in Apex post...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1626


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1626

[kenn]

This closes #1626


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b38c9e9e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b38c9e9e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b38c9e9e

Branch: refs/heads/master
Commit: b38c9e9ebc38ed4d39c18ab1e24256111c298a11
Parents: 4323247 8c00f63
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:34:48 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:34:48 2016 -0800

--
 .jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--