BlakeOrth commented on PR #18146: URL: https://github.com/apache/datafusion/pull/18146#issuecomment-3423790198
> @BlakeOrth what are your next planned steps? If I may suggest it, I think we should move ahead with adding a cache for the results of calling `ObjectStore::list`. I think we should be able to make everything faster (we don't have to choose) @alamb I think my ideal next step if this PR were to be merged would be adding a default implementation of the `ListFilesCache` that leverages the existing cache infrastructure. We could obviously do this _without_ merging this work, but doing so would leave directory-partitioned tables even further removed from flat tables. I suppose in theory we could first implement the default cache and _then_ work to normalize access patterns between the two (or manually add the cache somewhere into the partitioned workflow). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
