devinjdangelo commented on PR #9648: URL: https://github.com/apache/arrow-datafusion/pull/9648#issuecomment-2002483969
> It seems that `AbortableWrite` needs `MultiPart` information. 🤔 > > https://github.com/apache/arrow-datafusion/blob/37253e57beb25f0f1a4412b75421a489c2cb3c6a/datafusion/core/src/datasource/file_format/write/mod.rs#L155-L167 > > I'm reading the code to see whether `put_multipart` here can be replaced. We could drop the `AbortableWrite` construct entirely and no longer make a manually effort to abort on failure. The manual cleanup will not work in all cases anyway, and users can configure automatic cleanup with their cloud provider, e.g.: https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/. I believe this is in line with what @tustvold is thinking with ongoing improvements of `ObjectStore` https://github.com/apache/arrow-rs/pull/5500#discussion_r1522801072. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
