sunchao opened a new pull request, #6429: URL: https://github.com/apache/datafusion-comet/pull/6429
## Which issue does this PR close? No linked issue. ## Rationale for this change Native Parquet scan metrics count logical object-store GETs but cannot distinguish repeated storage reads from retries inside the S3 client. Two scans can both report 100 GETs even when one needed 105 HTTP attempts. Per-scan attempt counts help diagnose this difference when multiple scans share an S3 client. ## What changes are included in this PR? Add three counters to native execution metrics and the Spark SQL UI: `scan_io_http_observed_gets`, `scan_io_http_attempts`, and `scan_io_http_retries`. A request carries its scan's counters through HTTP-status retries and interrupted-body resumes. At a settled snapshot, attempts equal observed GETs plus retries. The connector forwards the existing client options, requests, responses, and errors. Counts cover calls at object_store's HTTP interface, excluding HEAD, credential requests, region discovery, and reqwest-internal redirects or protocol retries. The observed-GET counter distinguishes missing instrumentation coverage from a measured zero retry count. ## How are these changes tested? Added loopback HTTP tests for recovered and exhausted retries, interrupted-body resume, and concurrent scans sharing a client while only one retries. An integration test constructs S3 through Comet's production store builder and reads through the native Parquet reader factory, checking attribution to separate scans. Existing reader tests check that an uninstrumented store reports logical GETs but no observed HTTP attempts. A Spark test checks counter types and independent scan accumulators. The native Parquet test group passed: 246 tests passed, including all four HTTP regressions. Five existing tests were ignored: four require an HDFS cluster and one is a synthetic Variant benchmark. The run used the locked public dependencies and default native features. Full-reactor Spotless and changed-file Rust formatting checks passed. All 25 tests in `CometTaskMetricsSuite` passed with Spark 4.1.3 and JDK 21 in a root-reactor Maven run. The staged JNI library matched the freshly rebuilt native library byte-for-byte. Scala style checks and `git diff --check` passed. Instrumentation overhead has not been measured; no query performance improvement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
