Tushar7012 commented on issue #19971: URL: https://github.com/apache/datafusion/issues/19971#issuecomment-3801934957
## Rationale As discussed in #19971, [list_files_for_scan](cci:1://file:///d:/Agentic_AI/Gssoc_Apache/datafusion/datafusion/catalog-listing/src/table.rs:697:4-832:5) can be a bottleneck for tables with many files/partitions because it processes them sequentially (even with `try_join_all`, the CPU-heavy metadata parts ran on the same task). ## Changes - **Parallelization**: Refactored [list_files_for_scan](cci:1://file:///d:/Agentic_AI/Gssoc_Apache/datafusion/datafusion/catalog-listing/src/table.rs:697:4-832:5) in [datafusion/catalog-listing/src/table.rs](cci:7://file:///d:/Agentic_AI/Gssoc_Apache/datafusion/datafusion/catalog-listing/src/table.rs:0:0-0:0) to use `tokio::task::JoinSet`. This spawns a separate Tokio task for each partition listing, allowing metadata deserialization and statistics processing to run in parallel on multiple threads. - **Send Compliance**: Refactored `ListingTableUrl::list_prefixed_files` and related methods in [datafusion/datasource/src/url.rs](cci:7://file:///d:/Agentic_AI/Gssoc_Apache/datafusion/datafusion/datasource/src/url.rs:0:0-0:0) to accept `&ConfigOptions` and `&Arc<RuntimeEnv>` instead of `&dyn Session`. This ensures the arguments are `Send` and can be moved into the spawned tasks. - **Dependencies**: Added `tokio` dependency to `datafusion-catalog-listing`. ## Verification - `cargo test -p datafusion-catalog-listing` passed. - `cargo test -p datafusion-datasource` passed. - `cargo clippy` and `fmt` checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
