goldmedal opened a new issue, #10986: URL: https://github.com/apache/datafusion/issues/10986
### Is your feature request related to a problem or challenge? I had some discussions with @alamb about supporting a dynamic file data source (`select ... from 'select .. from 'data.parquet'` like #4805) in the core, as mentioned in https://github.com/apache/datafusion/issues/4850#issuecomment-2142190951. However, we found that it's not a good idea to move so many dependencies (e.g., S3-related) to the core crate after #10745. ### Describe the solution you'd like As @alamb proposed in https://github.com/apache/datafusion/pull/10745#issuecomment-2175817937, we can focus first on the logic that interprets table names as potential object store locations. Implement a struct `DynamicTableProvider` and a trait called `UrlLookup` to get `ObjectStore` at runtime. ```rust struct DynamicTableProvider { // ... /// A callback function that is url_lookup: Arc<dyn UrlLookup> } /// Trait for looking up the correct object store instance based on URL pub trait UrlLookup { fn lookup(&self, url: &Url) -> Result<Arc<dyn ObjectStore>>; } ``` By default, `DynamicTableProvider` only supports querying local file paths like `file:///...`. The implementation of dynamic file queries in datafusion-cli might also be based on `DynamicTableProvider` but will load the common object storage dependency by default. ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
