cetra3 opened a new pull request, #26407: URL: https://github.com/apache/datafusion/pull/26407
## Which issue does this PR close? - Related to #21968. There is no dedicated issue for this change. ## Rationale for this change When a `ListingTable` over Parquet files collects statistics, planning time grows with the number of files multiplied by the size of their footers, and none of that work runs in parallel. On wide tables it dominates planning. `ListingTable::collect_files_for_scan` gathers per-file statistics with `buffer_unordered(meta_fetch_concurrency)`. That makes the object store requests concurrent, but every future is polled by the one planning task. As a result, the footer decode and statistics conversion for each file run one file at a time, on whichever thread is polling the planner. In a profile of `wide_schema` Q02 (1024 columns, 256 files) with the metadata and statistics caches disabled, the planning thread spent about 345 ms per query, which is roughly the whole query time. About three quarters of that time was inside `ParquetFormat::infer_stats_and_ordering`. With the default cache limits the statistics cache cannot hold the statistics for this table, so the same work happens on every query. ## What changes are included in this PR? - `ParquetFormat::infer_stats_and_ordering` resolves its session-dependent inputs (the decryption properties and the metadata cache) on the calling task. When it runs inside a multi-thread Tokio runtime, it then fetches and decodes the footer, and computes the statistics and ordering, in a `SpawnedTask`. With the existing `buffer_unordered`, up to `meta_fetch_concurrency` files are decoded in parallel on the runtime's threads. - Outside a multi-thread runtime (a current-thread runtime, or no runtime at all, as in the wasm tests), it does the work on the calling task as before, so the caller's `buffer_unordered` polls it together with the other files. This is the same check `spawn_buffered` uses. - The shared work moves into a private helper, `fetch_stats_and_ordering`, which both paths call. This follows the convention described in the crate docs: CPU-bound work runs as tasks on the Tokio runtime, through `SpawnedTask`, rather than in `spawn_blocking`. Dropping a `SpawnedTask` aborts it, so when `get_files_with_limit` stops early, outstanding work is still cancelled. `ParquetFormat::infer_schema` has the same serial pattern when a table is created. It is left for a follow-up. ## What is the testing strategy for this PR? The results are unchanged; only where the work runs has changed, so the existing tests cover it. The `#[tokio::test]` unit tests run on a current-thread runtime and take the inline path, while the sqllogictests run on a multi-thread runtime and take the spawned path. - `datafusion-datasource-parquet` lib tests - `datafusion-catalog-listing` tests - `datafusion` `parquet_integration` tests and the core `datasource::listing` tests - the Parquet sqllogictests The performance change will be checked with the `wide_schema` benchmark. ## Are there any user-facing changes? There are no API changes. Planning that collects Parquet statistics uses more than one thread when it runs inside a multi-thread Tokio runtime, and behaves as before otherwise. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019C4oRqstvxH5CXTWnot8CD -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
