tvalentyn commented on code in PR #24435:
URL: https://github.com/apache/beam/pull/24435#discussion_r1119110307
##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2474,9 +2474,13 @@ def typed(transform):
# Capture in closure (avoiding capturing self).
args, kwargs = self.args, self.kwargs
+ #TODO: We need to add support for multi-window inputs to
CombineGlobally()
def inject_default(_, combined):
if combined:
- assert len(combined) == 1
+ if len(combined) > 1:
+ raise ValueError(
+ "Input from multiple simultaneous windows"
+ "isn't currently supported for CombineGlobally()")
Review Comment:
Apologies for slow response here.
> My understanding is that, as defined, CombineGlobally is intended to
return a single-element PCollection and I'm not sure how to interpret that in a
multi-window context.
My understanding is that CombineGlobally should work for a multi-window
context and result in combining down to 1 element per each key+window pair. If
you are interested in looking further into this problem, you can reproduce this
pipeline in Java SDK, which presumably doesn't have this bug in examine the
behavior, and/or get a second opinion. That being said, in my opinion the
"Input from multiple simultaneous windows isn't currently supported for
CombineGlobally()" may discourage users from using CombineGlobally but it's
some thing that we should fix instead.
##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2474,9 +2474,13 @@ def typed(transform):
# Capture in closure (avoiding capturing self).
args, kwargs = self.args, self.kwargs
+ #TODO: We need to add support for multi-window inputs to
CombineGlobally()
def inject_default(_, combined):
if combined:
- assert len(combined) == 1
+ if len(combined) > 1:
+ raise ValueError(
+ "Input from multiple simultaneous windows"
+ "isn't currently supported for CombineGlobally()")
Review Comment:
Apologies for slow response here.
> My understanding is that, as defined, CombineGlobally is intended to
return a single-element PCollection and I'm not sure how to interpret that in a
multi-window context.
My understanding is that CombineGlobally should work for a multi-window
context and result in combining down to 1 element per each key+window pair. If
you are interested in looking further into this problem, you can reproduce this
pipeline in Java SDK, which presumably doesn't have this bug in examine the
behavior, and/or get a second opinion. That being said, in my opinion the
"Input from multiple simultaneous windows isn't currently supported for
CombineGlobally()" may discourage users from using CombineGlobally but it's
something that we should fix instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
