mechgouki commented on pull request #2807: URL: https://github.com/apache/iceberg/pull/2807#issuecomment-930797701
> A few points I'd like to discuss: > > 1. around private vendor of catalog implementation > > I remember @rdblue you talked about the possibility of having a RESTful Catalog implementation to plugin, would that help this Dell use case? > > 1. around S3 SDK version > > I have been thinking recently a lot about the SDK version, and maybe we could consider reverting to v1, and Dell can contribute just a `S3Catalog` instead. > > The reason I am thinking about reverting to v1 is because of client side encryption support. V2 was promised to offer client side encryption this summer which would let v2 SDK have full functionality compatibility with v1 plus supposedly better performance, but the whole project was significantly delayed and won't be done until years later. So there is also an ask for adding S3 client side encryption from user side, for which the only way to achieve that is through reverting to v1. > > I think this version change could be done given the fact that nothing around AWS client is publicly exposed. Some work is needed to update documentation around the dependency jars to add. But if we see enough benefits in adopting S3-like private vendors by reintroducing V1, I think this seems to be the best way to go. > > @danielcweeks what do you think about the S3 SDK situation? > > @mechgouki if we reintroduce v1 SDK, do you think you still need the dell module, or could you just implement a `S3Catalog` instead in AWS module? @jackye1995 Basically we have 2 areas which Dell EMC features could benefit as below: (1)Append operation in additional of MPU, if the client has less local cache for large object( like edge), Dell EMC object service could help here. (2)Atomic rename. We have If-Match and If-None-Match semantic as we support strong consistency model (within one site) from the very beginning. So in order to support these 2 changes, we need change both in FileIO and S3Catalog, which we can not directly use AWS module while we could based on V1 SDK to extend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
