[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096726#comment-16096726
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3610


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095720#comment-16095720
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3610

[BEAM-2371] Use dehydration-insensitive APIs in ParDoEvaluatorFactory

R: @tgroh

Peeled off #3334 so it can be a trivial review. Ignore the base commit, 
which is #3601.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ParDoEvaluatorFactory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3610.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3610


commit eef50852ba5d5185207a79149393bd5a41a084f0
Author: Kenneth Knowles 
Date:   2017-07-20T03:58:36Z

Use RehydratedComponents for memoized rehydration

commit 4dd75f750f0e846b7c1a918e013ee259cdfb0614
Author: Kenneth Knowles 
Date:   2017-06-07T21:35:09Z

Use dehydration-insensitive APIs in ParDoEvaluatorFactory




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095716#comment-16095716
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3609

[BEAM-2371] Use dehydration-insensitive APIs in WindowEvaluatorFactory

R: @tgroh 

Peeled off #3334 so it can be a trivial review. Ignore the base commit, 
which is #3601.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam WindowEvaluatorFactory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3609.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3609


commit 6bf16792e779047243a29bd2af507958779a5ffc
Author: Kenneth Knowles 
Date:   2017-07-20T03:58:36Z

Use RehydratedComponents for memoized rehydration

commit d4923c9ddcc8dc836b7f9774cd23b882d7a5a341
Author: Kenneth Knowles 
Date:   2017-06-07T20:58:11Z

Use dehydration-insensitive APIs in WindowEvaluatorFactory




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081637#comment-16081637
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3536


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081600#comment-16081600
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3509


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081293#comment-16081293
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3536

[BEAM-2371] Use URNs, not Java classes, in immutability enforcements

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh peeled this off #3334 as I think it is separable.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-enforcements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3536


commit 311547aa561bb314a8fe743b6f4677a2eaaaca50
Author: Kenneth Knowles 
Date:   2017-07-10T22:25:11Z

Use URNs, not Java classes, in immutability enforcements




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076911#comment-16076911
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3509

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh

This ports the DirectRunner's pipeline replacements to work even after a 
pipeline has been serialized and then deserialized, hence somewhat SDK-agnostic.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3509


commit 515cd09b9017d13cebd5a4a161743c75a58455b5
Author: Kenneth Knowles 
Date:   2017-07-06T16:24:22Z

Rehydrate PCollections

commit e942c8edec6cbb80122f1bd966490d31f0d8ae9b
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit 670aaa0ce34bed78e4b35886cf6c84d324738188
Author: Kenneth Knowles 
Date:   2017-07-06T16:24:55Z

Include PCollection in rehydrated PCollectionView

commit b4d1636d109842aada19c7a6582a91f5f16a58cc
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs

commit 0f0412a520499ea4458575b23f184bd6f8d6fcea
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit adb44ad37f1b766f79fa864c67cdf00f25e94d51
Author: Kenneth Knowles 
Date:   2017-06-12T22:11:49Z

Fix misleading comment in TransformHierarchy

commit 312460ad41f2e858dae08605cdc126d217d3630d
Author: Kenneth Knowles 
Date:   2017-06-12T22:12:18Z

Fix null checks in TransformHierarchy




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075930#comment-16075930
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3330


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065916#comment-16065916
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3351


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051212#comment-16051212
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3352


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048215#comment-16048215
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3351

[BEAM-2371] Fix getAdditionInputs for SplittableParDo transforms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is a prerequisite for working with these through Runner API utilities.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam 
SplittableParDo-getAdditionalInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3351


commit 1bc64eba44d4cecff71f12f9647ec5a650ebc743
Author: Kenneth Knowles 
Date:   2017-06-13T17:00:09Z

Fix getAdditionInputs for SplittableParDo transforms




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048216#comment-16048216
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3352

[BEAM-2371] Fix getAdditionalInputs, etc, for DirectRunner stateful ParDo 
override

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is a prerequisite for working with these through Runner API utilities.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam 
StatefulParDo-getAdditionalInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3352


commit 013ea721a7f1849eec6b7f726ec0f66ed6f2578e
Author: Kenneth Knowles 
Date:   2017-06-12T23:31:32Z

Fix getAdditionalInputs, etc, for DirectRunner stateful ParDo override




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045351#comment-16045351
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3338


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045043#comment-16045043
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3338

[BEAM-2371] Port some DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

@tgroh this is pulled off of #3330, the bits that I don't think are waiting 
for side inputs so I think it could go green right away. It should also be an 
easier review with fewer distractions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides-easy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3338.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3338


commit e66724e69117961840ccc649a3d0ba723cc1c7dd
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit f77147a4642a7b3b7dfef0c00c752bdea07256fd
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit eeffa9d69a9159b36733920cee9bc843d6b74307
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit e35177b5f2ca9cbb6f7c08ee122b8d9fa15b6633
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043517#comment-16043517
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles reopened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit 22b06906159b1e74ee68b9a57073126129264885
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs

commit 8b4630d4a97638e902dc5db8bcd79f65b366e558
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit e141e4a48f321a04e2f4dd404ff650826da2f07c
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit ce657ab34c6f9b11e453d58981fb4183a1358b5f
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043348#comment-16043348
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3330


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043347#comment-16043347
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043249#comment-16043249
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3239


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033871#comment-16033871
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3280


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033459#comment-16033459
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3274


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032406#comment-16032406
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3274

[BEAM-2371] Write files translation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

The Java DirectRunner needs `WriteFiles` to be well-defined with a payload 
in order to detect (and provide) runner-specified sharding.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam WriteFiles-translation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3274


commit 2ee7d69ea6f130186b38ae21abc2ab06aee2c5a7
Author: Kenneth Knowles 
Date:   2017-05-26T23:06:23Z

Add WriteFilesPayload to Runner API well-known transforms

commit 0093cf5e53cbe6f3e6d61f91ea44d0d1b0f0f901
Author: Kenneth Knowles 
Date:   2017-05-26T23:06:57Z

Make WriteFiles config publicly readable

commit c4f28a539ef7acba8cd561a13f3dc32c507cc124
Author: Kenneth Knowles 
Date:   2017-05-30T21:43:25Z

Add WriteFiles translation




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026618#comment-16026618
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3239

[BEAM-2371] Port DirectRunner override matchers to language-agnostic 
construction APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-matchers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3239


commit 6342a36380d7a8f9f5990e680fce7c6c74f3d925
Author: Kenneth Knowles 
Date:   2017-05-25T13:28:45Z

Add splittability to ParDoPayload

commit 4dfb9ce6fd2252167d9923cff956b6731cc7bfba
Author: Kenneth Knowles 
Date:   2017-05-25T13:29:16Z

Auxiiliary URNs for SDF

commit fbe29f3e56c94d366ce8ebfb7576884acc022440
Author: Kenneth Knowles 
Date:   2017-05-25T13:29:36Z

Extract splittability via Runner API

commit cbc514e32f719188e581578b382cd33e32bc4d8a
Author: Kenneth Knowles 
Date:   2017-05-25T13:30:00Z

Some more somewhat standard URNs

A test-only primitive: TestStream
A deprecated primitive: CreateView
A composite that the DirectRunner overrides: WriteFiles

commit 843f4531a9860a5f31ea9c18314ce0daf0b723f2
Author: Kenneth Knowles 
Date:   2017-05-25T13:45:27Z

Add GroupByKeyTranslation

commit 14b83a447e2d847740eb5ad50b8115636bc5190d
Author: Kenneth Knowles 
Date:   2017-05-25T13:56:23Z

Add CreatePCollectionViewTranslation

commit a9744d11746d32c63e9ea9aba6145a29c336fc50
Author: Kenneth Knowles 
Date:   2017-05-25T13:56:32Z

Add WriteFilesTranslation

commit d2e99713e92bf30ee5dfd36ae7eff79b5a42d69a
Author: Kenneth Knowles 
Date:   2017-05-26T17:54:14Z

Add TestStreamTranslation

commit ead03b3e8eb792d42e10a97006be05c53cda949c
Author: Kenneth Knowles 
Date:   2017-05-25T13:29:09Z

Add Runner API oriented PTransformMatchers for DirectRunner overrides

commit 72cb839a42b3df07829e988d861984ebcb157457
Author: Kenneth Knowles 
Date:   2017-05-25T13:31:16Z

DirectRunner override matchers using Runner API




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)