KKcorps opened a new pull request, #20044: URL: https://github.com/apache/pinot/pull/20044
`S3PinotFS` sends multipart parts one at a time, so a large file uploads at single-stream speed. In our case that was a 14 GB upsert snapshot uploaded from a minion. This adds `multiPartUploadParallelism` (default `1`, which keeps today's sequential behaviour). With a higher value, parts go out on a bounded thread pool that lives only for that upload. Each part reads its own byte range of the file, so the SDK can reopen it on a retry instead of relying on a shared stream. Part uploads now also refresh credentials on 401 or 403, and only one refresh happens across parallel parts. A failed part cancels the remaining parts and aborts the upload. Tested with a new mock-based `S3PinotFSMultipartUploadTest` at parallelism 1, 4 and 32, which checks part order, byte ranges and abort on failure. The s3mock `testMultiPartUpload` now runs with parallelism 2. I could not run it locally because there is no Docker here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
