robertwb commented on a change in pull request #11270: [BEAM-9639][BEAM-9608]
Improvements for FnApiRunner
URL: https://github.com/apache/beam/pull/11270#discussion_r407670612
##########
File path:
sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py
##########
@@ -75,21 +75,27 @@
IMPULSE_BUFFER = b'impulse'
+# SideInputId is identified by a consumer ParDo + tag.
+SideInputId = Tuple[str, str]
+
+DataSideInput = Dict[SideInputId,
+ Tuple[bytes, beam_runner_api_pb2.FunctionSpec]]
+
class Stage(object):
"""A set of Transforms that can be sent to the worker for processing."""
def __init__(self,
name, # type: str
transforms, # type: List[beam_runner_api_pb2.PTransform]
- downstream_side_inputs=None, # type: Optional[FrozenSet[str]]
+ downstream_side_inputs=None, # type: Optional[Dict[str,
SideInputId]]
Review comment:
The goal of this (which, yes, should have been better documented) is to
quickly be able to prohibit fusion. But the reason we defined our own union was
so that memory didn't grow as O(n^2) in the common case because many stages
were able to share this set (rather than have their own copy). These changes
seem to break that.
Also, could you clarify why this was made into a dict?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
