dhruv-pratap opened a new pull request, #6019: URL: https://github.com/apache/iceberg/pull/6019
Context: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1666112648002419 We at Netflix here are trying to integrate PyIceberg 0.1.0 with our Iceberg Rest Catalog Service and realized there is a gap in the PyIceberg rest client that we need to address. For background, at Netflix all client-server interaction happens over TLS and is client side auth enforced for security purposes. The rest client that sits inside PyIceberg at present uses requests module for interaction with rest catalog service. Although this module allows the CA trust bundle to be [set via an environment variable](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification), but it does not allow a similar mechanism for setting [client side certificates](https://requests.readthedocs.io/en/latest/user/advanced/#client-side-certificates) via environment variable and has to be done programmatically when setting up a requests client. The below two approaches were discussed on the Slack thread: 1. RestCatalog exposes requests.Session() and accepts a pre-configured Session object during initialization. This puts the onus on the PyIceberg consumer to configure the session with correct auth mechanism, connection pooling, custom headers etc., and makes the RestCatalog client dumb and simply use the provided session to perform API interaction. The current RestSpec kind of assumes/dictates the auth mechanism to be oauth which might not be the case for every enterprise. 2. The other alternative is to define new set of PyIceberg properties and add to the [current PyIceberg configuration spec](https://py.iceberg.apache.org/#configuration) to accept SSL configuration. Example: catalog.rest.ssl.client.key catalog.rest.ssl.client.cert [catalog.rest.ssl.ca](http://catalog.rest.ssl.ca/) This would be minimal amount of changes, but I could see this list growing overtime with every new enterprise adoption requiring customization. After discussion with @samredai and @Fokko we agreed on approach #2, and this PR is to address the same. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
