Hi Jack, Daniel, We use several S3-compatible backends with Iceberg, these include S3, GCS, and others. Currently, S3FileIO provides us all the functionality we need Iceberg to talk to these backends. The way we create S3FileIO is via the constructor and provide the S3Client as the constructor param; we do not use the initialize(Map<String,String>) method in FileIO. Our custom catalog accepts the FileIO object at creation time. To talk to GCS, we create the S3Client with a few overrides (described below) and pass it to S3FileIO. After that, the rest of the S3FileIO code works as is. The only exception is that “gs” (used by GCS URIs) needs to be accepted as a valid S3 prefix. This is the reason I sent the email.
The reason why we want to use S3FileIO to talk to GCS is that S3FileIO almost works out of the box and contains all the functionality needed to talk to GCS. The only special requirement is the creation of the S3Client and allow “gs” prefix in the URIs. Based on our early experiments and benchmarks, S3FileIO provides all the functionality we need and performs well, so we didn’t see a need to create a native GCS FileIO. Iceberg operations that we need are create, drop, read and write objects from S3 and S3FileIO provides this functionality. We are managing ACLs (IAM in case of GCS) at the bucket level and that happens in our custom catalog. GCS has ACLs but IAMs are preferred. I’ve not experimented with ACLs or encryption with S3FileIO and that is a good question whether it works with GCS. But, if these features are not enabled via default settings, S3FileIO works just fine with GCS. I think there is a case for supporting S3-compatible backends in S3FileIO because a lot of the code is common. The question is whether we can cleanly expose the common S3FileIO code to work with these backends and separate out any specialization (if required) OR we want to have a different FileIO implementation for each of the other S3 compatible backends such as GCS? I’m eager to hear more from the community about this. I’m happy to discuss and follow long-term design direction of the Iceberg community. The S3Client for GCS is created as follows (currently the code is not open source so I’m sharing the steps only): 1. Create S3ClientBuilder. 2. Set GCS endpoint URI and region. 3. Set a credentials provider that returns null. You can set credentials here if you have static credentials. 4. Set ClientOverrideConfiguration with interceptors in the overrideConfiguration(). The interceptors are used to setup authorization header in requests (setting projectId, auth tokens, etc.) and do header translation for requests and responses. 5. Build the S3Client. 6. Pass the S3Client to S3FileIO. Thanks, Mayur From: Jack Ye <yezhao...@gmail.com> Sent: Wednesday, December 1, 2021 1:16 PM To: Iceberg Dev List <dev@iceberg.apache.org> Subject: Re: Supporting gs:// prefix in S3URI for Google Cloud S3 Storage Hi Mayur, I know many object storage services have allowed communication using the Amazon S3 client by implementing the same protocol, like recently the Dell EMC ECS and Aliyun OSS. But ultimately there are functionality differences that could be optimized with a native FileIO, and the 2 examples I listed before both contributed their own FileIO implementations to Iceberg recently. I would imagine some native S3 features like ACL or SSE to not work for GCS, and some GCS features to be not supported in S3FileIO, so I think a specific GCS FileIO would likely be better for GCS support in the long term. Could you describe how you configure S3FileIO to talk to GCS? Do you need to override the S3 endpoint or have any other configurations? And I am not an expert of GCS, do you see using S3FileIO for GCS as a feasible long-term solution? Are there any GCS specific features that you might need and could not be done through S3FileIO, and how widely used are those features? Best, Jack Ye On Wed, Dec 1, 2021 at 8:50 AM Daniel Weeks <daniel.c.we...@gmail.com<mailto:daniel.c.we...@gmail.com>> wrote: The S3FileIO does use the AWS S3 V2 Client libraries and while there appears to be some level of compatibility, it's not clear to me how far that currently extends (some AWS features like encryption, IAM, etc. may not have full support). I think it's great that there may be a path for more native GCS FileIO support, but it might be a little early to rename the classes and except that everything will work cleanly. Thanks for pointing this out, Mayur. It's really an interesting development. -Dan On Wed, Dec 1, 2021 at 8:12 AM Piotr Findeisen <pi...@starburstdata.com<mailto:pi...@starburstdata.com>> wrote: if S3FileIO is supposed to be used with other file systems, we should consider proper class renames. just my 2c On Wed, Dec 1, 2021 at 5:07 PM Mayur Srivastava <mayur.srivast...@twosigma.com<mailto:mayur.srivast...@twosigma.com>> wrote: Hi, We are using S3FileIO to talk to the GCS backend. GCS URIs are compatible with the AWS S3 SDKs and if they are added to the list of supported prefixes, they work with S3FileIO. Thanks, Mayur From: Piotr Findeisen <pi...@starburstdata.com<mailto:pi...@starburstdata.com>> Sent: Wednesday, December 1, 2021 10:58 AM To: Iceberg Dev List <dev@iceberg.apache.org<mailto:dev@iceberg.apache.org>> Subject: Re: Supporting gs:// prefix in S3URI for Google Cloud S3 Storage Hi Just curious. S3URI seems aws s3-specific. What would be the goal of using S3URI with google cloud storage urls? what problem are we solving? PF On Wed, Dec 1, 2021 at 4:56 PM Russell Spitzer <russell.spit...@gmail.com<mailto:russell.spit...@gmail.com>> wrote: Sounds reasonable to me if they are compatible On Wed, Dec 1, 2021 at 8:27 AM Mayur Srivastava <mayur.srivast...@twosigma.com<mailto:mayur.srivast...@twosigma.com>> wrote: Hi, We have URIs starting with gs:// representing objects on GCS. Currently, S3URI doesn’t support gs:// prefix (see https://github.com/apache/iceberg/blob/master/aws/src/main/java/org/apache/iceberg/aws/s3/S3URI.java#L41). Is there an existing JIRA for supporting this? Any objections to add “gs” to the list of S3 prefixes? Thanks, Mayur
