adriangb opened a new pull request, #27110: URL: https://github.com/apache/datafusion/pull/27110
> [!NOTE] > Draft prototype stacked on https://github.com/apache/datafusion/pull/27022. Only the last two commits are new; review those. ## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/26682 - Prototype for the suggestion in https://github.com/apache/datafusion/pull/27022#pullrequestreview-5776318105 ## Rationale for this change The I/O tests in https://github.com/apache/datafusion/pull/27022 only see object store requests. They cannot say which scan step caused a request. The review suggested `tracing::debug!` events at important places, so tests can snapshot planning and scan operations as well as I/O. ## What changes are included in this PR? 1. `datafusion-datasource-parquet` depends on `tracing` (no default features). The scan emits spans and events at these places: - opening a file - loading metadata, with metadata cache hits and misses - loading the page index and the Bloom filters, with one event per Bloom filter read - row group pruning with statistics and with Bloom filters, and page index pruning - column chunk fetches and row filter evaluation 2. Each call site records its own fields. The library formats no strings for tests. 3. The test harness (`core/tests/parquet/io/trace_capture.rs`) is a `tracing_subscriber` layer. It renders spans and events as an indented tree and prints every recorded field. The test object store emits a span for every operation and names the Parquet structure that each byte range covers. The tests look like this: ```rust let test = filter_pushdown_test(true).await; let traces = test.capture_traces(); test.query("SELECT * FROM t WHERE val = 7").await.unwrap(); assert_snapshot!(traces.render(), @" Query (sql=SELECT * FROM t WHERE val = 7, rows=1) ... Fetch column chunk data (ranges=1) ObjectStore::get_ranges (location=table.parquet, ranges=1) Read byte range (range=145..192, parquet=dictionary page and data page 0 for column 'val' (rg 1)) Evaluate row filter (predicate=val@0 = 7, rows=3, rows_matched=1) ... "); ``` Two findings came out of this: - In https://github.com/apache/datafusion/pull/27022, "Request 1: Read Parquet footer (offsets 0..678)" is really the listing table's `HEAD` request. The store reports a `GET` with `head: true` as a read of the whole file. - Planning loads the page index into the cached metadata before the scan opens the file. ## What is the testing strategy for this PR? The 14 snapshot tests in `datafusion/core/tests/parquet/io/` were converted to the new API and regenerated with `cargo insta accept`. The capture uses `tracing::subscriber::set_default`, which only applies to the current thread. This works because `#[tokio::test]` runs every task on the test thread. ## Are there any user-facing changes? There is a new `tracing` dependency in `datafusion-datasource-parquet`. Events are at `debug` level and cost almost nothing when no subscriber is installed. The API is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
