alexanderbianchi opened a new pull request, #27167: URL: https://github.com/apache/datafusion/pull/27167
## Which issue does this PR close? No linked issue; draft API proposal. ## Rationale for this change Applications embedding DataFusion can use a common scan boundary to collect application-defined metrics and attach tracing context across data sources. This adds an optional hook around `DataSourceExec` partition execution, with metric names, labels, and export policy owned by the application. ## What changes are included in this PR? - A `DataSourceInstrumentation` session extension receives the source, partition, task context, metrics set, and a callback that opens the partition. - The hook can observe opening errors and wrap the output stream to observe batches, errors, completion, and cancellation. Output is observed after source limits and batch splitting. - Hook-created metrics are retained by `DataSourceExec` and exposed alongside source metrics, including for sources without their own persistent metric set. Resetting the plan or replacing its source starts a fresh set. - Documentation and a runnable registration example. ## What is the testing strategy for this PR? Public API integration tests cover memory and CSV scans, batch splitting, labeled metrics, execution-time registration, reset and limit behavior, opening and stream errors, and cancellation. A doctest covers registration and a row-counting stream wrapper. Local validation passed: - `cargo test --profile ci -p datafusion --test core_integration datasource` (37 tests) - `cargo test --profile ci -p datafusion-datasource --doc DataSourceInstrumentationExtension` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo fmt --all --check` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` The optional full `dev/rust_lint.sh` suite was not completed; it was stopped during installation of additional lint tools. ## Are there any user-facing changes? An additive, opt-in API in `datafusion-datasource`. The extension is selected from the executing session and introduces no serialized plan fields. It applies to sources executed through `DataSourceExec`; applications executing plans in other processes install the extension in those sessions as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
