reidkaufmann opened a new issue, #889: URL: https://github.com/apache/arrow-rs-object-store/issues/889
**Describe the bug** - [`LocalUpload::complete`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L1144-L1145) takes `self.src` before publishing the staging file. When `metadata`, `sync_all`, `close` or the rename then fails, the error is returned but `<dest>#N` stays on disk: [`abort`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L1172-L1173) returns `Aborted` and [`Drop`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L1183-L1193) does nothing, both because `self.src` is already `None`. `list` hides `#N` files and a retry picks `#N+1`; the leaked file costs disk space and keeps its directory non-empty, which stops automatic cleanup from removing the directory. - [`put_opts`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L443-L446) and [`copy_opts`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L654-L658) remove their staging file on any error from `rename()`, including a directory fsync that fails after the rename succeeded. At that point the staging path is free again and the next `put`, `put_multipart` or `copy` to the same destination picks the first free `#N`, so the removed file may be a new upload's. **To Reproduce** `put_multipart`, one `put_part`, make the destination a non-empty directory so the rename fails: `complete()` returns `Err` and `<dest>#1` remains (test `multipart_complete_failure_removes_staging_file` in the linked PR). The second bug needs a directory fsync that fails after a successful rename and was not reproduced. **Expected behavior** A failed publish removes the staging file when it fails before the rename succeeds. After a successful rename the staging path may already belong to a new upload, so a later failure leaves it alone and only reports the error. The rule is asymmetric because the rename is the point where the staging path stops belonging to this upload. **AI usage** Written with AI assistance; I reviewed it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
