DerGut opened a new pull request, #3667: URL: https://github.com/apache/iceberg-rust/pull/3667
## Which issue does this PR close? This PR is another attempt at closing https://github.com/apache/iceberg-rust/issues/1466. It supersedes my original https://github.com/apache/iceberg-rust/pull/1496 and is the first of three planned PRs. The idea of what's to come: 1. (this PR): Scan report collection and lifecycle, with a logging consumer 2. Public reporter API and configuration 3. REST catalog scan reporting ## What changes are included in this PR? This first PR implements the collection of metrics relevant for scan reports. It counts number of manifests, data files, total bytes, etc. It adds a (for now private and synchronous) reporter trait to capture the counts in tests. A follow-up PR will work on its public API. The commits are ordered for independent review: 1. [`a5dc8d131` — join-handle lifecycle](https://github.com/DerGut/iceberg-rust/commit/a5dc8d131): retain and join scan plan workers, so we can wait for their results. 4. [`fead59434` — counting](https://github.com/DerGut/iceberg-rust/commit/fead59434): collect counts throughout the scan planning stages. 5. [`b5769a3d2` — reporting](https://github.com/DerGut/iceberg-rust/commit/b5769a3d2): finalize and deliver completed reports through the private `MetricsReporter`. ## Are these changes tested? The Java comparison harness on branch [`scan-reports-1-validation`](https://github.com/DerGut/iceberg-rust/tree/scan-reports-1-validation/dev/scan-reports) runs both implementations against the same metadata, manifests, and physical files. It compares file/delete plans, report fields, counters, and units. The separate branch contains pinned revisions/toolchains, commands, and [machine-readable results](https://github.com/DerGut/iceberg-rust/blob/scan-reports-1-validation/dev/scan-reports/evidence/comparison.json). The test branch is not meant to be merged because of it's setup that doesn't fit well into the repo's existing integration test suite. The 22 fixtures cover: - Manifest/file pruning, dead entries and dead-only manifests, and zero-result scans, including cases using statistics produced by the Parquet writer. - Positional deletes attached to zero, one, or multiple data files; equality-delete predicate pruning and bounds-based associations; partition type promotion. - Deletion-vector content bytes and supersession of positional deletes, plus delete-manifest pruning. - Case-insensitive selection, current-schema/time-travel behavior, nested select-all, and metadata-column projections. Eighteen fixtures match all compared fields and plans. Four retain explained differences: _partial-projection schema identity_, _nested projection IDs_, _equality-delete predicate pruning_, and _equality-delete association_. The latter two expose existing planner differences: Rust attaches deletes that Java can exclude using column bounds, and the counters reflect those different plans. The harness does not normalize these differences away or claim blanket Java parity. ## AI Disclosure AI assisted implementation and validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
