Thanks, Tarun. In addition to boot.go, SDK is also looking for some portion of the staged files, like pickled_main_session <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> (somehow we are still using it after cloudpickle migration for new purposes), in #1/#2 SDK would need to be aware that there is an assigned worker ID (and it is not the same ID anymore as the ID within boot.go, due to sibling SDK protocol <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> ).
We could do a targeted fix just for requirements.txt, which is even more narrow and reversible. Though I think the proper way to solve this is on the runner. I understand there is some urgency to have a quick fix as multi-env pipelines might be blocked. If those pipelines are blocked already, do we have a workaround for affected users, for example to use a custom container instead of passing --requirements_file that collides in Python vs YAML environment? If we have a workaround, I'd suggest to try to prototype a runner side fix before rushing in the SDK change. I think fixing the contract on the runner is a cleaner direction (see #A vs #B in my earlier email), though others are welcome to weigh in here. If there are reasons to believe #B is the way to go (SDK must deduplicate), then I'd explore if we can strengthen some guarantees for files supplied via --files_to_stage (perhaps via a proto change), so that we determine the location for these files in a more deterministic manner, possilbly irrespective of the runner.
