behcetm commented on issue #40420: URL: https://github.com/apache/arrow/issues/40420#issuecomment-1996794094
Hi @phofl, Thanks for bringing up this topic. This limitation of the fixed size parts also prevents uploading objects larger than 100GB which I think is quite low compared to the maximum object size(5 TB) supported by AWS S3. I would be very interested for a solution to this issue. My preferred solution would be having the part size configurable per output stream. I guess that could still work for Cloudflare R2. Behcet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
