Abacn commented on issue #30505: URL: https://github.com/apache/beam/issues/30505#issuecomment-2959810372
There are a few options 1, the simplest - Instead of putting harness jar at the front of class path, we can put it at the end: https://github.com/apache/beam/blob/b56f99db5b0fabc43bf278cb3abf8a8e9f10d7e9/sdks/java/container/boot.go#L163 - make Dataflow runner having runtime dependency on SDK harness Then user submitting pipeline will usually stage SDK harness, that overwrites the one embedded in SDK container 2. We can change our PostRelease test to overwrite worker jar / SDK harnss by adding `--experiments=use_staged_dataflow_worker_jar` when run on SNAPSHOT Beam version It includes the following change: In https://github.com/apache/beam/blob/b56f99db5b0fabc43bf278cb3abf8a8e9f10d7e9/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml#L202, add ``` <snapshots> <dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-harness</artifactId> <version>${beam-version}</version> </dependency> <dependency> <groupId>org.apache.beam</groupId> <artifactId><artifactId>beam-runners-google-cloud-dataflow-java-legacy-worker</artifactId></artifactId> <version>${beam-version}</version> </dependency> </snapshots> ``` and in https://github.com/apache/beam/blob/b56f99db5b0fabc43bf278cb3abf8a8e9f10d7e9/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L2582 add ``` if (ver.endsWith('SNAPSHOT' && config.runner == 'DataflowRunner') { argsNeeded.add("--experiment=use_staged_dataflow_worker_jar") } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
