Hi everyone, New findings and discussions are available at go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>. While the high level idea is to align Python dependency management similar to Java(previous work <https://github.com/apache/beam/issues/20009>). We realized that some Python use cases assumed a fixed path existed. For example --files_to_stage <https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/>, which is used by run_inference <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116>. With this proposal the SEMI_PERSISTENT_DIRECTORY is no longer a static path, known during pipeline submission. So, this would break their pipelines; they should apply a similar fix (go/beam-pr/40795 <https://goto.google.com/beam-pr/40795>).
Here are a few options: 1. [Quick Fix]: Narrow down the scope of problem. Instead of moving all the dependencies into separate environment-based paths. We narrow down(hardcode) specific files(example <https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L71>, which we already do) This will allow us to solve this issue in the latest release cut (planned today). 2. [SDK fix]: Instead of listing all these files manually in the boot.go. Stager marks these files with a separate marker in the pipeline proto. The boot process downloads the files into separate directories (shared vs. environment-based) 3.[Runner fix]: Runner can host different folder, mounted at the same path in each container(host /var/opt/google/<container> mounted at /var/opt/google) So, there is no conflict between the containers. May be with this, we can remove environment-based downloads for Java as well. It looks clean, but it will increase worker memory, as each container will now download and store dependencies separately. With all these options, customers using the run_inference <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116> pattern are still un-impacted. I recommend option-1, to unblock multi-env YAML pipelines, option-3 needs agreement on the memory trade-off. Thank you, Tarun. On Wed, Oct 21, 2026 at 11:14 AM Valentyn Tymofieiev <[email protected]> wrote: > +1, thank you! > > On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev < > [email protected]> wrote: > >> +1. Thanks a lot for looking into this and providing a fix Tarun. >> >> - Cham >> >> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <[email protected]> wrote: >> >>> Nice catch, and good write-up. Thanks for the clear summary in the doc, >>> too. >>> >>> Kenn >>> >>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev < >>> [email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> Recently, we have encountered Dataflow Multi-Environment pipelines >>>> crash-looping due to integrity check failures. However, while debugging we >>>> realized it's a long-standing issue in Beam Python dependency staging. This >>>> fix is small, but a bit convoluted, so I have created a doc and attached >>>> the PR. >>>> >>>> Doc: >>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing >>>> >>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699> >>>> >>>> Thank you, >>>> Tarun. >>>> >>>
