rdblue commented on pull request #30141: URL: https://github.com/apache/spark/pull/30141#issuecomment-715449287
I think what we ended up doing was to generate a UUID in `OutputCommitter.setupJob` and place it in the Hadoop `Configuration`. Now that the configuration is reliable for a given stage, wouldn't that work here instead of using the one from Spark? I'm okay adding this. It matches what we do in v2 writes. The new API also passes a UUID in, so I think there is a reasonable precedent for it, even if the committer or writer could generate one itself. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
