stczwd commented on pull request #28280: URL: https://github.com/apache/spark/pull/28280#issuecomment-700645391
> is it possible that a task knows it fails to commit and clean up the files? Em, I have thought about this. This is a good way to remove extra output file and can also be used in `FileFormatWriter.write`. However, it will still left temporary directory uncleaned, as each task only cleaned their output dir, none of them will clean the temporary dir. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
