Sajish85 commented on issue #17202: URL: https://github.com/apache/iceberg/issues/17202#issuecomment-7060555274
Adding measurements that may be useful here. I ran a fault-injection study of Iceberg table maintenance (rewrite_data_files, expire_snapshots, remove_orphan_files) against a live ingest, killing or pausing maintenance at chosen phases and checking the table afterwards against a ledger of acknowledged writes, with an independent second reader (pyiceberg). 369 cells across 1.7.1, 1.12.0 and main (2d89aa4). On the behaviour reported in this issue: in the study's race cells the signature appeared in 8 of 8 cells on 1.7.1. Because the overlap there is timing-dependent, I also ran a gated variant that forces the expire commit to land inside the rewrite's commit window; under that forced overlap the signature appeared in 3 of 3 cells on 1.7.1, 8 of 8 on 1.12.0 and 7 of 8 on main. In every flagged cell the reported counts matched what committed; the only misleading field was failed_data_files_count = 0. I understand the current behaviour is considered working as designed for partial progress; this is offered as reproduction data on current builds, including a checker that can be re-run: paper https://doi.org/10.5281/zenodo.24412833, code and per-cell results https://github.com/Sajish85/iceberg-maintenance-fault-injection. Happy to adjust the harness if a different configuration is the one that matters. — Sajish Krishnan (independent researcher) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
