[
https://issues.apache.org/jira/browse/BEAM-8402?focusedWorklogId=333634&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333634
]
ASF GitHub Bot logged work on BEAM-8402:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Oct/19 18:16
Start Date: 24/Oct/19 18:16
Worklog Time Spent: 10m
Work Description: robertwb commented on issue #9811: [BEAM-8402] Create a
class hierarchy to represent environments
URL: https://github.com/apache/beam/pull/9811#issuecomment-546039607
I bring this up because I think it's important to at least have some idea of
the long-term story before we start introducing the explicit notion of
environments to the public API. (I think there's still the open question on
what the intended public API and usecase is, which would be informative in
helping guide this story.)
One of the problems is this: if I have a pipeline with an environment
"embedded" for a DoFn it now is tied to a particular runner (currently, either
a Java runner or a Python runner). This is contrary to the idea that a pipeline
proto should be runner-agnostic, and in practice comes up when trying to use an
expansion service that asks for the embedded-java environment on the Python
direct runner, or the Python embedded environment when running on Flink. And
neither work on Dataflow.
As for the safety of asking for "Java with Beam 2.16.0" and swapping I think
that such an ask should be interpreted as the vanilla container--rather than
building a custom container with the same name as the Java default one should
be building a custom container with its own name.
I agree that fleshing out the idea completely could take some time, but if
we have the basic shape we can leave some details TBD while still having a
forward-looking API .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 333634)
Time Spent: 1h 20m (was: 1h 10m)
> Create a class hierarchy to represent environments
> --------------------------------------------------
>
> Key: BEAM-8402
> URL: https://issues.apache.org/jira/browse/BEAM-8402
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Chad Dombrova
> Assignee: Chad Dombrova
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As a first step towards making it possible to assign different environments
> to sections of a pipeline, we first need to expose environment classes to the
> pipeline API. Unlike PTransforms, PCollections, Coders, and Windowings,
> environments exists solely in the portability framework as protobuf objects.
> By creating a hierarchy of "native" classes that represent the various
> environment types -- external, docker, process, etc -- users will be able to
> instantiate these and assign them to parts of the pipeline. The assignment
> portion will be covered in a follow-up issue/PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
