kennknowles commented on code in PR #29140:
URL: https://github.com/apache/beam/pull/29140#discussion_r1372280103
##########
model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_expansion_api.proto:
##########
@@ -53,6 +54,10 @@ message ExpansionRequest {
// coders for the output PCollections. Note that the request
// may not be fulfilled.
map<string, string> output_coder_requests = 4;
+
+ // (Optional) A set of Pipeline Options that should be used
+ // when expanding this transform.
+ google.protobuf.Struct pipeline_options = 5;
Review Comment:
@chamikaramj just checking this with you
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java:
##########
@@ -140,4 +140,12 @@ static String getEnvironmentOption(
boolean getEnableHeapDumps();
void setEnableHeapDumps(boolean enableHeapDumps);
+
+ @Description(
+ "If set, attempts to produce a pipeline compatible with this prior
version of the Beam SDK."
+ + " See
https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline.";)
Review Comment:
I think we actually have some discussions on dev@ and this also is Flink and
Samza. Makes sense, basically. It isn't rigorously defined, but we do OK with
it. And I think it is more or less runner independent.
##########
sdks/python/apache_beam/runners/portability/expansion_service.py:
##########
@@ -41,7 +43,18 @@ def __init__(self, options=None):
def Expand(self, request, context=None):
try:
- pipeline = beam_pipeline.Pipeline(options=self._options)
+ options = copy.deep_copy(self._options)
+ request_options = pipeline_options.PipelineOptions.from_runner_api(
+ request.pipeline_options)
+ # TODO(https://github.com/apache/beam/issues/20090): Figure out the
+ # correct subset of options to apply to expansion.
+ if request_options.view_as(
Review Comment:
same comment re: being not just for GCP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
