danielcweeks opened a new pull request #1767: URL: https://github.com/apache/iceberg/pull/1767
Add progressive upload to S3OutputStream using multipart upload. A few key changes are: 1. The output stream will switch from single PUT based upload to multipart when the multipart size * threshold is reached. 2. Staging files will be rotated and uploaded async as output written 3. Staging files will be deleted when the upload completes to reduce local disk requirements 4. An attempt will be made to abort the upload upon failure, but this is not guaranteed (Bucket lifecycle rules should be implemented to ensure data is cleaned up.) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
