[
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025898#comment-16025898
]
ASF GitHub Bot commented on BEAM-115:
-------------------------------------
GitHub user kennknowles opened a pull request:
https://github.com/apache/beam/pull/3233
[BEAM-115] Runner API Translations for StateSpec and TimerSpec
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [x] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`.
- [x] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [x] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
---
R: @tgroh
The awkward bit is that a combining `StateSpec` is statically typed to take
a `CombineFn`. So when we get around to issuing Fn State API requests, we'll
need to work around that.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kennknowles/beam translate-StateSpec
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3233.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3233
----
commit c87b72d5522e3369becd7fbe022824f4f223a9ae
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-25T14:12:08Z
Flesh out TimerSpec and StateSpec in Runner API
commit 55af992a3a465b333bc3ae51262bfc43474d90e8
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-25T14:25:08Z
Mark CombineFnWithContext StateSpecs internal
commit 34eca25c5a6b6f3733f51fb6cb421cc755b7058c
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-25T14:27:52Z
Add case dispatch to StateSpec
This is different than a StateBinder: for a binder, the id is needed and
the StateSpec controls the return type. For case dispatch, the
dispatcher controls the type and it should just be reading the spec,
which does not require the id. Eventually, StateBinder could be removed
in favor of StateSpec.Cases<Function<String, StateT>>.
commit 09aeab25a92ef961a8968a5a3e863786750dff46
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-25T20:02:15Z
Allow translation to throw IOException
commit 9eb1ef070506e7b419bd388f2a0f9407056a8bcb
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-26T05:51:18Z
Make Java serialized CombineFn URN public
commit 5aa01d64ff50d9396137ada84b281700ce1d8d8d
Author: Kenneth Knowles <[email protected]>
Date: 2017-05-25T14:12:29Z
Implement TimerSpec and StateSpec translation
----
> Beam Runner API
> ---------------
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
> Issue Type: Improvement
> Components: beam-model-runner-api
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical
> vision.
> It has technical limitations:
> - The user's DAG (even including library expansions) is never explicitly
> represented, so it cannot be analyzed except incrementally, and cannot
> necessarily be reconstructed (for example, to display it!).
> - The flattened DAG of just primitive transforms isn't well-suited for
> display or transform override.
> - The TransformHierarchy isn't well-suited for optimizations.
> - The user must realistically pre-commit to a runner, and its configuration
> (batch vs streaming) prior to graph construction, since the runner will be
> modifying the graph as it is built.
> - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from
> actual cases of failure to use according to the design)
> - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner
> is confusing.
> - The TransformHierarchy, accessible only via visitor traversals, is
> cumbersome.
> - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus,
> and building an API that more simply and directly supports the technical
> vision.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
