claudevdm commented on PR #34324: URL: https://github.com/apache/beam/pull/34324#issuecomment-2730390239
> Thanks @claudevdm. I know there is context somewhere, but could you briefly explain why we need this change here? Looks like we are now getting the pane index in a separate and new DoFn prior to Reshuffle, rather than from the original DoFn. The Reshuffle implementation in python does not preserve pane index (see https://github.com/apache/beam/issues/28219) If we keep fetching pane info in the original DoFn then ALL load jobs will have pane info 0 (see linked bug) and only the very first load job will succeed. So another option is to fix the Reshuffle implementation (I have tested this too), but that is a more intrusive change that probably needs more discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
