[
https://issues.apache.org/jira/browse/BEAM-6049?focusedWorklogId=166500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-166500
]
ASF GitHub Bot logged work on BEAM-6049:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Nov/18 16:39
Start Date: 15/Nov/18 16:39
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #7047:
[BEAM-6049] Add option to load job to GCS in Dataflow Runner
URL: https://github.com/apache/beam/pull/7047#discussion_r233906127
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -196,6 +193,7 @@
@VisibleForTesting static final int GCS_UPLOAD_BUFFER_SIZE_BYTES_DEFAULT =
1024 * 1024;
@VisibleForTesting static final String PIPELINE_FILE_NAME = "pipeline.pb";
+ @VisibleForTesting static final String DATAFLOW_GRAPH_FILE_NAME =
"dataflow_graph.pb";
Review comment:
I'm curious how much of the information in `dataflow_graph.pb` is not
contained within `pipeline.pb`. As long as Dataflow Service is building the
functionality to read the graph from a file, would it be better to consume the
`pipeline.pb` directly and work towards deprecating the Dataflow-specific
representation at this layer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 166500)
Time Spent: 0.5h (was: 20m)
> Support Job / Graph upload to GCS in Dataflow Runner
> ----------------------------------------------------
>
> Key: BEAM-6049
> URL: https://issues.apache.org/jira/browse/BEAM-6049
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Andrea Foegler
> Assignee: Andrea Foegler
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Add uploadGraph flag to support uploading the job / graph to GCS instead of
> embedding in the request. This change allows the API to support much larger
> graphs than the embedded format of CreateJob.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
