[
https://issues.apache.org/jira/browse/BEAM-8402?focusedWorklogId=336468&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336468
]
ASF GitHub Bot logged work on BEAM-8402:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Oct/19 21:09
Start Date: 30/Oct/19 21:09
Worklog Time Spent: 10m
Work Description: violalyu commented on issue #9811: [BEAM-8402] Create a
class hierarchy to represent environments
URL: https://github.com/apache/beam/pull/9811#issuecomment-548113132
> I bring this up because I think it's important to at least have some idea
of the long-term story before we start introducing the explicit notion of
environments to the public API. (I think there's still the open question on
what the intended public API and usecase is, which would be informative in
helping guide this story.)
Hi @robertwb, here's an example of how it might be used:
```python
pipe = beam.Pipeline(options=options)
init = pipe | beam.Create(some_iterable)
with pipe.environment(DockerEnvironment(container_image='dummy_registry')):
section_1 = (
init
| beam.ParDo(UserDefinedDoFn())
| # arbitrary number of operations here
| 'furtherProcess' >> beam.Map(lambda x: x)
)
with pipe.environment(DockerEnvironment(container_image='another_registry')):
section_2 = (
section_1
| beam.ParDo(AnotherUDF())
)
with pipe.environment(ExternalEnvironment(url='localhost:50000')):
section_3 = (
section_1
| beam.ParDo(ExternalUDF())
)
...
```
where `pipe.environment` acts like a context manager which will assign the
given environment to all transforms inside it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 336468)
Time Spent: 1h 50m (was: 1h 40m)
> Create a class hierarchy to represent environments
> --------------------------------------------------
>
> Key: BEAM-8402
> URL: https://issues.apache.org/jira/browse/BEAM-8402
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Chad Dombrova
> Assignee: Chad Dombrova
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> As a first step towards making it possible to assign different environments
> to sections of a pipeline, we first need to expose environment classes to the
> pipeline API. Unlike PTransforms, PCollections, Coders, and Windowings,
> environments exists solely in the portability framework as protobuf objects.
> By creating a hierarchy of "native" classes that represent the various
> environment types -- external, docker, process, etc -- users will be able to
> instantiate these and assign them to parts of the pipeline. The assignment
> portion will be covered in a follow-up issue/PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
