lwfitzgerald opened a new pull request, #2543: URL: https://github.com/apache/iceberg-python/pull/2543
# Rationale for this change The existing S3 remote signing hook function (`s3v4_rest_signer`) uses `requests.post` to submit `POST` requests to the REST signing endpoint. This internally creates a new `requests.Session` for every request, preventing any reuse of connections. In my profiling I saw this add overhead from repeated loading of CA certs and reestablishing of TLS connections. This change makes the signing function a callable object that wraps a `request.Session`, using this for `POST`ing, therefore achieving connection reuse. Signer callables are stored on the hook internals of the `aiobotocore` client inside the `s3fs.S3FileSystem` instance, so use and lifetime will match that of those instances. They are thread-local since: https://github.com/apache/iceberg-python/pull/2495. ## Are these changes tested? Tested locally. Existing unit tests updated for changes. ## Are there any user-facing changes? Yes - S3 signing requests now use connection pools (tied to the `s3fs.S3FileSystem` object). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
