robertwb commented on a change in pull request #11270:
URL: https://github.com/apache/beam/pull/11270#discussion_r412384268
##########
File path:
sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -75,21 +75,27 @@
IMPULSE_BUFFER = b'impulse'
+# SideInputId is identified by a consumer ParDo + tag.
+SideInputId = Tuple[str, str]
+
+DataSideInput = Dict[SideInputId,
+ Tuple[bytes, beam_runner_api_pb2.FunctionSpec]]
+
class Stage(object):
"""A set of Transforms that can be sent to the worker for processing."""
def __init__(self,
name, # type: str
transforms, # type: List[beam_runner_api_pb2.PTransform]
- downstream_side_inputs=None, # type: Optional[FrozenSet[str]]
+ downstream_side_inputs=None, # type: Optional[Dict[str,
SideInputId]]
Review comment:
Discussed offline, but capturing here for the record. These sets contain
the transitive collection of everything downstream of any side-input consuming
transform, and as such can be large even if the total number of side inputs is
small. (The number of distinct such sets is about the same as the number of
side inputs, so we keep the total memory use down by re-using them--to give
each transform its own copy would easily be O(n^2).)
Your change of computing the side input mapping after the graph has been
fused is good (and arguably better, as you only need the immediate consumers,
and don't have to re-compute each time a stage is fused).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
