kennknowles commented on PR #28272: URL: https://github.com/apache/beam/pull/28272#issuecomment-1716775279
Right. I was, indeed, thinking of reshuffle via random key. But what I'm saying does apply to Reshuffle by assigned keys. I have been a person who always says that Reshuffle is a semantic noop. If we should preserve that we need the v2 and the SDK implementation to record the pane info (aka reify `WindowedValue` instead of `PaneIndex`). This issue has shaken up my beliefs. If you have `Combine -> (stage) -> Reshuffle`. Each input to reshuffle has a pane index that corresponds to its key+window in the combine. The main thing is that if you have the original key+window+index you can reassemble the aggregate and/or write it to a sink in a consistent way. But if the stage changes the key you lose that property. Do we even have a good story for what happens to pane info when a DoFn reassigns to new keys? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
