[
https://issues.apache.org/jira/browse/BEAM-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535090#comment-17535090
]
Kenneth Knowles commented on BEAM-14403:
----------------------------------------
A while ago we had a similar problem with {{PTransform.expand}} methods doing
different things based on the pipeline options, even though they were not
supposed to. Many even checked what the runner was and built a totally
different graph based on the runner, which meant they ended up not being truly
portable, or not actually implemented in terms of the Beam model. We solved
this problem by completely removing access to this information, so PTransforms
had no choice but to have a portable expansion.
An analogous solution might be to have service options exist outside the
pipeline options. Or to have pipeline options that are somehow not actually
available in the code paths where the runner is submitting things.
I want to note that the issue here is in the {{DataflowRunner}} not the core
SDK. I know many people do not care about that distinction. But the issue is in
the logic that decides how to submit a Beam graph to Dataflow. So it is not
really a generalized problem anyhow. It is a small enough code base that code
review, and a general best practice of having as few options as possible,
should suffice.
> Allow Prime to be used with Legacy workers
> ------------------------------------------
>
> Key: BEAM-14403
> URL: https://issues.apache.org/jira/browse/BEAM-14403
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.35.0, 2.36.0, 2.37.0, 2.38.0
> Reporter: Robert Bradshaw
> Assignee: Robert Bradshaw
> Priority: P1
> Fix For: 2.39.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/v2.37.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L2315]
> assumes that enable_prime forces UW. This decision should be left up to the
> service.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
