BlakeOrth commented on issue #19273: URL: https://github.com/apache/datafusion/issues/19273#issuecomment-3643187556
@Yuvraj-cyborg I think you generally have the idea right! > modifying the get_with_extra method to accept the table's base path as the Extra parameter In my mind this should be the other way around. I think `key` should always be the table's base path, and `extra` should be the prefix. This way `key` remains the same between `get` and `get_with_extra` and the prefix becomes an optional addition to a cache lookup. This also generally remains consistent with how the actual listing calls are structured here: https://github.com/apache/datafusion/blob/bde16083ad344b7a52db5cb298a15d7434ffde51/datafusion/datasource/src/url.rs#L238 This should also allow us to (internally) always call `get_with_extra` from `get` to avoid some code duplication. Using the above makes the cache lookup subtly different from what you've described, where a lookup for the `key` will always return the entry for the table (if it exists), and then the contents `extra` can be used to further filter the results returned by the cache. So to use your example: `get_with_extra(key="my_table", extra="a=1")` 1. Fetch the cache entry for `my_table` 2. Discard any entries that do not match `my_table/a=1` 3. Return filtered results without making a storage call -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
