yihua opened a new issue, #988:
URL: https://github.com/apache/hudi-rs/issues/988

   `Storage::new` calls `object_store::parse_url_opts` every time, and one 
table open, plan and read reaches it from several places: table properties, the 
timeline, the file system view, the metadata table open and its reader, and 
each file group reader. Each call builds a new HTTP client, connection pool and 
credential provider, so on cloud storage every one of them opens its own 
connections and fetches credentials again (on EC2, from the instance metadata 
service).
   
   A table's `Storage` instances that point at the same bucket with the same 
options could share one object store for the life of the table, so connections 
and cached credentials are reused across the open, the plan and the reads. 
Sharing should stay within one table: a store holds resolved credentials, so it 
should not outlive or leave the table that supplied them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to