[
https://issues.apache.org/jira/browse/BEAM-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897400#comment-16897400
]
Chamikara Jayalath commented on BEAM-7850:
------------------------------------------
Thanks Max. I understand that this will be a bit invasive change. But I think
it will simplify implementations quite a bit. So the main motivation is
implementation simplicity. Currently to determine the environment where a given
PTransform has to be executed, we have to fork for all possible payload types.
Just to clarify, if we make Environment a top level attribute of Transform that
will be enough for WindowInto transform as well, right ?
Looking at Beam runner API proto, seems like only two messages that use
SdkFunctionSpec and are not a payload of a certain type of PTransform are
WindowingStrategy and SideInput. SideInput already seems to be within transform
payload messages (ParDoPayload and WriteFilesPayload). We can keep environment
around in SdkFunctionSpec if there are non-transform messages that need it.
If we consider the cross-language expansion scenario, what we get from a remote
environment is an expanded PTransform (along with coders, inputs, outputs,
dependencies needed to execute it). So I think it makes sense to associate a
PTransform with an Environment directly.
> Make Environment a top level attribute of PTransform
> ----------------------------------------------------
>
> Key: BEAM-7850
> URL: https://issues.apache.org/jira/browse/BEAM-7850
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
> Reporter: Chamikara Jayalath
> Priority: Major
>
> Currently Environment is not a top level attribute of the PTransform (of
> runner API proto).
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
> Instead it is hidden inside various payload objects. For example, for ParDo,
> environment will be inside SdkFunctionSpec of ParDoPayload.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
>
> This makes tracking environment of different types of PTransforms harder and
> we have to fork code (on the type of PTransform) to extract the Environment
> where the PTransform should be executed. It will probably be simpler to just
> make Environment a top level attribute of PTransform.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
