jackye1995 commented on pull request #2807: URL: https://github.com/apache/iceberg/pull/2807#issuecomment-930842682
had some offline discussion with @mechgouki . Here are some conclusions: 1. we can make this as the `S3Catalog` implementation in AWS module. In the future even if other vendors including AWS S3 come up with similar semantics, we can add a catalog config to switch across implementations. 2. we can add a catalog config like `use-append` to switch to the append output stream implementation in `S3FileIO`. Overall the Netflix `S3OutputStream` seems to be still more performant, but the use case around append-optimized object storage looks like a reasonable use case to support. 3. these all depends on the switch to v1 SDK, but I think it seems to be mutually benefical given that v2 does not support custom header for third-party vendors. Reverting to v1 can provide Iceberg more vendor integration, more features, and reduce the number of modules we need to create for new vendors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
