tvalentyn commented on a change in pull request #13303:
URL: https://github.com/apache/beam/pull/13303#discussion_r522675942
##########
File path:
sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -1212,6 +1213,22 @@ def get_ancestors(name):
return None
+def _parent_for_fused_stages(stages, context):
+ # type: (Iterable[str], TransformContext) -> Optional[str]
+
+ '''Returns the name of the new parent for the fused stages.
+
+ The new parent is the lowest common ancestor of the fused stages that is not
+ contained in the set of fused stages. The provided context is used to compute
+ ancestors of stages.
+ '''
+ result = functools.reduce(
+ lambda a, b: _lowest_common_ancestor(a, b, context), stages)
+ if result in stages:
+ result = context.parents_map().get(result)
Review comment:
is it guaranteed that `result in stages` no longer holds after this?
##########
File path:
sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -788,14 +789,14 @@ def get_stage_key(stage):
pcoll_id_remap = {}
remaining_stages = []
for sibling_stages in grouped_eligible_stages.values():
+ if len(sibling_stages) == 1:
+ ineligible_stages.extend(sibling_stages)
Review comment:
dumb question: what's a key-with-none stage? Is it obvious why a
single-sibling stage should be marked 'ineligible' or it needs a comment?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
