tustvold commented on issue #831: URL: https://github.com/apache/arrow-rs-object-store/issues/831#issuecomment-5389162716
For context https://github.com/apache/arrow-rs/pull/5500 contains the original design intent for the trait and some of the reasons for the current design. The big challenge is providing a mechanism that works for LocalFileystem without forcing double-buffering. This is why part IDs are not exposed, as they don't actually mean anything for LocalFileystem. That being said I do remember an early design where UploadPart was actually a stream that you could poll to keep retrying, but eventually that got scrapped for being too complex, as it made the trait very unwieldy. I suspect providing customisable retry policies is probably the simplest solution here. Whilst MultipartStore gives full control, LocalFileystem can't implement it. An alternative might be to somehow extend MultipartUpload with a method that returns a retryable version of UploadPart, but that would almost certainly be a breaking change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
