chamikaramj commented on code in PR #22230:
URL: https://github.com/apache/beam/pull/22230#discussion_r920422821
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java:
##########
@@ -898,8 +898,19 @@ protected RunnerApi.Pipeline applySdkEnvironmentOverrides(
throw new RuntimeException("Error parsing environment docker
payload.", e);
}
String containerImage = dockerPayload.getContainerImage();
+ boolean updated = false;
for (int i = 0; i < overrides.length; i += 2) {
containerImage = containerImage.replaceAll(overrides[i], overrides[i
+ 1]);
+ if (!containerImage.equals(dockerPayload.getContainerImage())) {
+ updated = true;
+ }
+ }
+ if (containerImage.startsWith("apache/beam")
Review Comment:
It's in DataflowRunner.java so technically this is a part of the runner. We
override the container during job submission for Dataflow (not during
expansion).
> We should definitely provide a docker hub mirror for official beam images
on gcr and configure it as a default container registry instead of external
docker hub
If we want to do this, I would consider this and orthogonal change, not a
blocker for x-lang releases. I think it took weeks last time Emily tried to
perform such a migration. For now I think we should override the container base
to what Dataflow prefers at job submission. We already do this for all
non-x-lang Java pipelines and all Python pipelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
