[
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867182#comment-15867182
]
ASF GitHub Bot commented on BEAM-115:
-------------------------------------
GitHub user kennknowles opened a pull request:
https://github.com/apache/beam/pull/2011
[BEAM-115,BEAM-1348] Unify Fn API and Runner API coder specs
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [x] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [x] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [x] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
This includes the entirety of #2000, which unified `FunctionSpec`. Feel
free to review that first or, if you prefer, review just this since the diff is
small.
The unification here was uninteresting on top of #2000. Summary of changes:
- Moved a coder's local id out of the `Coder` itself and into the key of a
map in `ProcessBundleDescriptor`. Philosophically, the id is an essential
aspect of `ProcessBundleDescriptor` (or `Pipeline`) but not not an essential
aspect of a coder. Pragmatically, this allows the Runner API and the Fn API to
key the map on different types (`string` and `int64` respectively).
Prospectively, it makes it easy to construct instances of the message that are
"just values" without any id, which is aesthetically pleasing and more flexible
to more uses.
- Inlined the `SdkFunctionSpec` in `Coder` in the Runner API. Having it by
reference introduces a needless sharing of key type and adds needless overhead,
since coders are already stored by reference, as are environments.
R: @dhalperi
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kennknowles/beam fn-api-coders
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/2011.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2011
----
commit 2b55a7f303ab0fea58ad279dd214253b4fe69565
Author: Kenneth Knowles <k...@google.com>
Date: 2017-02-14T20:33:43Z
Remove underscore from Runner API proto Java package
commit 4e7865b828eae962532f1759833eed8b0e769cc9
Author: Kenneth Knowles <k...@google.com>
Date: 2017-02-13T16:38:40Z
Unify Fn API and Runner API FunctionSpec
commit 5b5e6290e893385c47799cf5523c29be64c102fd
Author: Kenneth Knowles <k...@google.com>
Date: 2017-02-15T03:51:58Z
Unify Fn API and Runner API coder spec
----
> Beam Runner API
> ---------------
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
> Issue Type: Improvement
> Components: beam-model-runner-api
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical
> vision.
> It has technical limitations:
> - The user's DAG (even including library expansions) is never explicitly
> represented, so it cannot be analyzed except incrementally, and cannot
> necessarily be reconstructed (for example, to display it!).
> - The flattened DAG of just primitive transforms isn't well-suited for
> display or transform override.
> - The TransformHierarchy isn't well-suited for optimizations.
> - The user must realistically pre-commit to a runner, and its configuration
> (batch vs streaming) prior to graph construction, since the runner will be
> modifying the graph as it is built.
> - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from
> actual cases of failure to use according to the design)
> - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner
> is confusing.
> - The TransformHierarchy, accessible only via visitor traversals, is
> cumbersome.
> - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus,
> and building an API that more simply and directly supports the technical
> vision.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
