robertwb commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1097866767
##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
return element
+def _extract_py_version(capability):
+ return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+ # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+ py_envs_at_submission = set()
+ for _, env in process_bundle_descriptor.environments.items():
Review Comment:
> thanks for feedback. i am also thinking to stop checking as soon as one
environment successfully matched,
We should require that all the environments (referenced in this bundles
transforms) match.
> and also considering doing the matches at most once in process lifetime.
let me know if you think it's over-optimizing.
We need to repeatedly raise the error to get the pipeline to fail. This
happens once per (already expensive and rare) bundle processor initialization,
so I wouldn't worry about trying to limit it to once per process (though a
stronger case could be made about warnings about package version mismatch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
