yihua opened a new issue, #988: URL: https://github.com/apache/hudi-rs/issues/988
`Storage::new` calls `object_store::parse_url_opts` every time, and one table open, plan and read reaches it from several places: table properties, the timeline, the file system view, the metadata table open and its reader, and each file group reader. Each call builds a new HTTP client, connection pool and credential provider, so on cloud storage every one of them opens its own connections and fetches credentials again (on EC2, from the instance metadata service). A table's `Storage` instances that point at the same bucket with the same options could share one object store for the life of the table, so connections and cached credentials are reused across the open, the plan and the reads. Sharing should stay within one table: a store holds resolved credentials, so it should not outlive or leave the table that supplied them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
