Sajish85 commented on issue #17202:
URL: https://github.com/apache/iceberg/issues/17202#issuecomment-7060555274

   Adding measurements that may be useful here. I ran a fault-injection study 
of Iceberg table maintenance (rewrite_data_files, expire_snapshots, 
remove_orphan_files) against a live ingest, killing or pausing maintenance at 
chosen phases and checking the table afterwards against a ledger of 
acknowledged writes, with an independent second reader (pyiceberg). 369 cells 
across 1.7.1, 1.12.0 and main (2d89aa4).
   
   On the behaviour reported in this issue: in the study's race cells the 
signature appeared in 8 of 8 cells on 1.7.1. Because the overlap there is 
timing-dependent, I also ran a gated variant that forces the expire commit to 
land inside the rewrite's commit window; under that forced overlap the 
signature appeared in 3 of 3 cells on 1.7.1, 8 of 8 on 1.12.0 and 7 of 8 on 
main. In every flagged cell the reported counts matched what committed; the 
only misleading field was failed_data_files_count = 0.
   
   I understand the current behaviour is considered working as designed for 
partial progress; this is offered as reproduction data on current builds, 
including a checker that can be re-run: paper 
https://doi.org/10.5281/zenodo.24412833, code and per-cell results 
https://github.com/Sajish85/iceberg-maintenance-fault-injection. Happy to 
adjust the harness if a different configuration is the one that matters.
   
   — Sajish Krishnan (independent researcher)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to