Thomas Groh created BEAM-1721: --------------------------------- Summary: Reshuffle can shift elements in time Key: BEAM-1721 URL: https://issues.apache.org/jira/browse/BEAM-1721 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Thomas Groh Assignee: Thomas Groh
The reshuffle transform is meant to have no visible effects on the data that it processes. However, due to the use of a {{GroupByKey}}, the timestamp of the output elements is determined by the {{OutputTimeFn}} of the input {{WindowingStrategy}} Elements should not be shifted in time when being processed in {{Reshuffle}}. Currently this would require reifying all timestamps before applying the GroupByKey and reapplying them after. As an intermediate solution, elements should never be shifted forwards in time, as doing so permits the watermark to advance improperly (if the elements already contain their timestamps, for example), and prevents the timestamps from being reassigned within a {{DoFn}} or via the {{WithTimestamps}} transform. -- This message was sent by Atlassian JIRA (v6.3.15#6346)