ZhangqyTJ commented on issue #2136: URL: https://github.com/apache/arrow-datafusion/issues/2136#issuecomment-1090009486
> Hi @ZhangqyTJ, my perspective for the object store for a standalone system is that we should leverage the _ObjectStoreRegistry_ as a system-level one rather than the session-level one currently implemented. And we should not do the manually registration for object stores. Instead, we should register them by default. Therefore, we proposed #2111 to introduce object stores as optional features in the datafusion core. For HDFS as an example, the detailed implementation will load related configurations from configuration files under specific locations, which may also work for s3. For example, to put and load your configurations from ~/.datafusion/object_stores/s3.json. @matthewmturner > > In the future, we should refine the current implementation from the following aspects: > > * Extract the _ObjectStoreRegistry_ as a system-level property for both of the Scheduler and the Executor. > * Send the object store related configurations from the Scheduler to the Executors when doing the executor registration so that we don't need to put configuration files to every executor node. ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
