rdettai opened a new issue #1187: URL: https://github.com/apache/arrow-datafusion/issues/1187
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The ability to read partitioned tables is added the the `ListingTable` in #1141. In that implementation the table lists all the files before applying the partition pruning. This will be very slow for large tables. **Describe the solution you'd like** Instead of calling `collect()` on the stream of all files: - if a filter exists on the first level of partitions, first call the `list_dir()` feature of the object store to get the first level of partition and apply the filter on it - if the filter above was applied and was "selective enough", only list the files in the resulting folders for further pruning - otherwise use `list_file` on the entire table but evaluate the pruning progressively to stop listing as soon as limit is reached. **Describe alternatives you've considered** The current implementation works well on reasonably sized tables (few thousand files), but will fall short on huge tables (e.g. 100k file). **Additional context** The function where the magic happens: - https://github.com/rdettai/arrow-datafusion/blob/b3b32d4f56d2ac5df932749b98a4b2e95ac08d47/datafusion/src/datasource/listing/helpers.rs#L158-L261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
