Thanks everyone, for the suggestions! I updated the doc to accommodate the new design we discussed on the PR go/beam-pr/40699 <https://goto.google.com/beam-pr/40699>
https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing Thank you, Tarun. On Thu, Oct 29, 2026 at 6:09 PM Valentyn Tymofieiev <[email protected]> wrote: > It looks like there is an experiment that can be used to temporarily > override the sha checks while service side mitigation is in progress. This > is not ideal, and we still have a race condition during artifact > materialization in the latest SDK release. But this race is a preexisting > problem. Therefore my preference would be not to incur technical debt in > the SDK with a rushed workaround if we can avoid it, so I am voting -0 on > implementing option 1 in the ongoing release. > > I think the existing workaround (opt-out experiment for hash validations) > can be acceptable until the next release since the experiment can be > user-controlled, and this is probably safer than landing a change to > artifact staging path done right before the release (bigger blast radius is > something goes wrong). If we find new information/opinion/evidence that > resolving staging conflict must be done in SDK rather than runner, then we > can create a better mechanism (e.g. option 1 or variation thereof) for the > next release. > > Thanks, > Valentyn > > On Thu, Oct 29, 2026 at 3:22 PM Chamikara Jayalath <[email protected]> > wrote: > >> >> >> On Thu, Oct 29, 2026 at 10:09 AM Tarun Annapareddy < >> [email protected]> wrote: >> >>> Unless there are objections. I am proposing the option-1 short-term fix >>> to unblock the 2.78 release cut. I will update the design doc for option-3, >>> once 2.78 is done, it also needs alignment from Runner. >>> >> >> Using option 1 to unblock the release sounds good to me. We can look into >> options 2 or 3 as the long term fix subsequently. >> >> >>> >>> Thank you, >>> Tarun. >>> >>> On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> ` I understand there is some urgency to have a quick fix as multi-env >>>>> pipelines might be blocked. If those pipelines are blocked already, do we >>>>> have a workaround for affected users, for example to use a custom >>>>> container >>>>> instead of passing --requirements_file that collides in Python vs YAML >>>>> environment?` >>>>> >>>>> > I do not have a good workaround for these customers. Multi-env YAML >>>>> pipelines will fail. I am surprised there is no report, perhaps because >>>>> the >>>>> regression started only in 2.76 and multi-env is not a regular use case. >>>>> >>>> >>>> This will include all Beam YAML users that use Python providers. >>>> >>>> https://beam.apache.org/documentation/sdks/yaml-providers/#python >>>> >>>> >>>>> >>>>> Thank you, >>>>> Tarun. >>>>> >>>>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks, Tarun. >>>>>> >>>>>> In addition to boot.go, SDK is also looking for some portion of the >>>>>> staged files, like pickled_main_session >>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> >>>>>> (somehow >>>>>> we are still using it after cloudpickle migration for new purposes), in >>>>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and >>>>>> it is not the same ID anymore as the ID within boot.go, due to sibling >>>>>> SDK protocol >>>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> >>>>>> ). >>>>>> >>>>>> We could do a targeted fix just for requirements.txt, which is even >>>>>> more narrow and reversible. Though I think the proper way to solve >>>>>> this is on the runner. I understand there is some urgency to have a >>>>>> quick fix as multi-env pipelines might be blocked. If those pipelines are >>>>>> blocked already, do we have a workaround for affected users, for example >>>>>> to >>>>>> use a custom container instead of passing --requirements_file that >>>>>> collides >>>>>> in Python vs YAML environment? >>>>>> >>>>>> If we have a workaround, I'd suggest to try to prototype a runner >>>>>> side fix before rushing in the SDK change. >>>>>> >>>>>> I think fixing the contract on the runner is a cleaner direction (see >>>>>> #A vs #B in my earlier email), though others are welcome to weigh in >>>>>> here. >>>>>> >>>>>> If there are reasons to believe #B is the way to go (SDK must >>>>>> deduplicate), then I'd explore if we can strengthen some guarantees for >>>>>> files supplied via --files_to_stage (perhaps via a proto change), so that >>>>>> we determine the location for these files in a more deterministic manner, >>>>>> possilbly irrespective of the runner. >>>>>> >>>>>
