[
https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857433#comment-15857433
]
ASF GitHub Bot commented on BEAM-802:
-------------------------------------
GitHub user mariapython opened a pull request:
https://github.com/apache/beam/pull/1945
[BEAM-802] Add ValueProvider class for FileBasedSource I/O Transforms
- [x] Add ValueProvider class.
- [x] Derive StaticValueProvider and RuntimeValueProvider from
ValueProvider.
- [x] Derive ValueProviderArgumentParser from argparse.ArgumentParser as
API for the template user.
- [x] Modify FileBasedSource I/O transforms to accept objects of type
ValueProvider.
- [x] Modify display_data.
- [x] Handle serialization / deserialization.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mariapython/incubator-beam ppp_inmaster
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1945.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1945
----
commit 3c04d39d8badcd841894fa6fef78b1888395e273
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-18T06:57:22Z
Add RuntimeValueProvider class
commit 0636f3cfa8d299e605aeb00f6a6f3d5a000d6877
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-27T19:24:01Z
Modify IO tests to accept StaticValueProvider objects.
commit cac9b3410a0ba8469b084d606d88ee60c7a09ba9
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-27T21:55:31Z
Add module to test ValueProvider
commit dd85305d4d4d67a6c7ba271a366a64881cd5c88e
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-27T22:00:15Z
Fix bug + clean code
commit 3600f8dcd0ceddf43db0f65c3e5af771189766df
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-27T22:02:34Z
Add serialization for StaticValueProvider
commit cdc53bb479bec67474a511646e778f8a74ccb376
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-28T18:42:11Z
Remove unnecessary optionsid from pipeline_options and pipeline_options_test
commit f817990537437e7fa7d22031171bf99cf519078f
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-28T18:46:31Z
Fix bug and correct typos in filebasedsource.py
commit 169b7ec58fb188db6915da1428323ee014b1ddf6
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-28T21:18:20Z
Disable validation for RuntimeValueProviders
commit 3d420dd40b7bee69bf7cdcf558c59735d50e2fb5
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-01-28T21:19:09Z
Add test to have FileBasedSource accept only str or ValueProvider
commit df0ac1f93703e758d502bbcfb634bc98560b732f
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-02-06T19:57:40Z
Have FileSink accept ValueProvider
commit f0dab8f9006177139b906b9cfed808f015cf6c8d
Author: Maria Garcia Herrero <[email protected]>
Date: 2017-02-08T05:19:05Z
Fix errors introduced from python-sdk --> master merge
----
> Support Dynamic PipelineOptions for python
> ------------------------------------------
>
> Key: BEAM-802
> URL: https://issues.apache.org/jira/browse/BEAM-802
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py
> Reporter: María GH
> Assignee: María GH
> Priority: Minor
> Original Estimate: 1,680h
> Remaining Estimate: 1,680h
>
> Goal: Enable users to run pipelines from templates filled via CL (pipeline
> options)
> Background: Currently, the Runner creates the JSON pipeline description which
> can be sent to the worker as is, since everything is already defined there
> (with links to gs:// for input and binaries). With the parametrized approach,
> those descriptions are empty and filled by the user or defaulted, so the
> pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that
> needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic
> key/value pairs provided at runtime (dynamic)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
