DerGut opened a new pull request, #3732:
URL: https://github.com/apache/iceberg-rust/pull/3732

   ## Which issue does this PR close?
   
   Works toward #1466.
   
   ## What changes are included in this PR?
   
   Implements scan metrics for the simple case. This means it instruments the 
number of manifest, data and delete files read/ skipped, the number of bytes 
selected and more (16 counters - the one's that Java also sends for parity).
   
   There are some edge-cases about early stops in scan planning that I'm 
deliberately leaving out of this PR to keep it small. Will follow-up on it. 
Also, the wiring into the public API from 
https://github.com/apache/iceberg-rust/pull/3709 will come in a follow-up.
   The full scan reporting implementation is a bunch of code unfortunately and 
I think this break-down allows for independently reviewable & independently 
verifiable chunks.
   
   ## Are these changes tested?
   
   Of course with unit tests. But more interestingly, the separate 
[Java-comparison 
branch](https://github.com/DerGut/iceberg-rust/tree/scan-planning-counters-java/dev/scan-reports)
 contains a harness that runs 39 test scenarios against this implementation and 
Java's and compares their reported scan metrics:
   
   - 35/39 scenarios match exactly
   - 2/39 scenarios mismatch because of equality-delete planner differences
   - 2/39 scenarios mismatch because early-close scheduling differences
   
   This PR deliberately focuses on the first 35. A follow-up will make the 
early-close differences match.
   
   ## AI Disclosure
   
   Yep, used AI for implementing and testing this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to