David Volquartz Lebech created BEAM-1648:
--------------------------------------------
Summary: Replace gsutil calls with Cloud Storage API
Key: BEAM-1648
URL: https://issues.apache.org/jira/browse/BEAM-1648
Project: Beam
Issue Type: Wish
Components: runner-dataflow, sdk-py
Affects Versions: 0.5.0
Reporter: David Volquartz Lebech
Assignee: Davor Bonaci
When using the {{DataflowRunner}} and {{--setup-file}} parameter, {{gsutil}} is
used for _some_ of the Cloud Storage uploads
[here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L89-L90].
This makes it difficult to run a pipeline in an environment where the Cloud
Platform tools are not installed -- e.g. a Docker Python container or a Heroku
instance.
The Storage API is used in other places such as
[here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L431-L432]
when staging the session and main SDK package, so I'm unsure if this is by
design or an inconsistency in the usage.
Thank you for considering this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
