Thomas Groh created BEAM-1721:
---------------------------------
Summary: Reshuffle can shift elements in time
Key: BEAM-1721
URL: https://issues.apache.org/jira/browse/BEAM-1721
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh
The reshuffle transform is meant to have no visible effects on the data that it
processes. However, due to the use of a {{GroupByKey}}, the timestamp of the
output elements is determined by the {{OutputTimeFn}} of the input
{{WindowingStrategy}}
Elements should not be shifted in time when being processed in {{Reshuffle}}.
Currently this would require reifying all timestamps before applying the
GroupByKey and reapplying them after. As an intermediate solution, elements
should never be shifted forwards in time, as doing so permits the watermark to
advance improperly (if the elements already contain their timestamps, for
example), and prevents the timestamps from being reassigned within a {{DoFn}}
or via the {{WithTimestamps}} transform.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
