DDtKey commented on issue #7436: URL: https://github.com/apache/arrow-datafusion/issues/7436#issuecomment-1697643304
That's definitely right point to avoid staged files. I think `ObjectStore` should create staged files as a temporary files (probably just hidden ones) with `Drop` logic to delete them in case they weren't completed. I don't think DF should care of the way `object_store` handles the staging. If a file write/upload wasn't completed - intermediate files should be cleaned up. Probably this may be configurable behavior WDYT? cc @tustvold @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
