[
https://issues.apache.org/jira/browse/BEAM-11719?focusedWorklogId=557668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557668
]
ASF GitHub Bot logged work on BEAM-11719:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Feb/21 01:54
Start Date: 25/Feb/21 01:54
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #13839:
URL: https://github.com/apache/beam/pull/13839#discussion_r582433711
##########
File path: sdks/python/apache_beam/runners/pipeline_context.py
##########
@@ -233,12 +233,20 @@ def requirements(self):
# rather than an actual coder. The element type is required for some runners,
# as well as performing a round-trip through protos.
# TODO(BEAM-2717): Remove once this is no longer needed.
- def coder_id_from_element_type(self, element_type):
- # type: (Any) -> str
+ def coder_id_from_element_type(
+ self, element_type, requires_deterministic_key_coder=None):
+ # type: (Any, Optional[str]) -> str
if self.use_fake_coders:
return pickler.dumps(element_type).decode('ascii')
else:
- return self.coders.get_id(coders.registry.get_coder(element_type))
+ coder = coders.registry.get_coder(element_type)
+ if requires_deterministic_key_coder:
+ coder = coders.TupleCoder([
Review comment:
Got it, requires_deterministic_key_coder is set by the ForceKvInputTypes
visitor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 557668)
Time Spent: 5h 40m (was: 5.5h)
> Enforce deterministic coding for GroupByKey and Stateful DoFns
> --------------------------------------------------------------
>
> Key: BEAM-11719
> URL: https://issues.apache.org/jira/browse/BEAM-11719
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Robert Bradshaw
> Assignee: Robert Bradshaw
> Priority: P1
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> If a non-deterministic coder, such as pickling, is used for keys this can
> result in two copies of the same key being grouped separately (based on their
> encodings).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
