jiengup opened a new issue, #4252: URL: https://github.com/apache/iggy/issues/4252
### Bug description During destructive partition state transfer, `install_backup::begin_with_storage` freezes the live materialization by hard-linking its files into `.install-building`. For each hard link, `link_tree` opens the destination through a new file description and calls `sync()` on that fresh handle before publishing `.install-backup`. On Linux, asynchronous writeback errors are tracked with per-open-file `errseq` cursors. If writeback fails after the original writer was opened but before the backup handle is opened, the backup handle samples past that error and can report a successful sync. The original writer still reports the failure. Expected: backup publication must fail unless every file that can be restored from the backup has completed a durability barrier through its original writer. A writeback error must fence the install before `.install-backup` is published. Actual: the fresh hard-link handle can report success after dirty bytes were dropped. `begin_with_storage` can then publish the backup and state transfer can durably unlink the live segment chain, leaving an inadequately fenced backup as the recovery source. This was found while auditing the related fresh-handle checkpoint problem in #4141, whose simulator coverage was introduced by #4132. Relevant code: - [`install_backup::link_tree`](https://github.com/apache/iggy/blob/ad8951f57fd1ba3641425a67f8ee61b85e91678b/core/partitions/src/install_backup.rs#L109-L149) - [`install_backup::begin_with_storage`](https://github.com/apache/iggy/blob/ad8951f57fd1ba3641425a67f8ee61b85e91678b/core/partitions/src/install_backup.rs#L69-L83) - [State transfer drains persistence and begins the backup](https://github.com/apache/iggy/blob/ad8951f57fd1ba3641425a67f8ee61b85e91678b/core/partitions/src/state_transfer.rs#L2701-L2719) - [The destructive live-materialization mutation](https://github.com/apache/iggy/blob/ad8951f57fd1ba3641425a67f8ee61b85e91678b/core/partitions/src/state_transfer.rs#L2902-L2949) ### Affected area / component Iggy server, Clustering / replication ### Deployment Not applicable ### Versions Current `master` at `ad8951f57fd1ba3641425a67f8ee61b85e91678b`. ### Hardware / environment The deterministic `SimStorage` model reproduces Linux per-file-description writeback error semantics. The production risk applies to Linux filesystems that report delayed writeback failures such as `EIO` or `ENOSPC`. ### Reproduction A deterministic regression can reproduce the window without real storage failure injection: ```rust let storage = storage_for_partition().await; let path = Path::new("/partition/materialized"); let mut writer = storage.open(path, OpenMode::Create).await.unwrap(); writer.write(0, b"pending".to_vec()).await.unwrap(); storage.sync_directory(Path::new("/partition")).await.unwrap(); storage.fail_writeback(path).unwrap(); let result = install_backup::begin_with_storage(Path::new("/partition"), &storage).await; assert!(writer.sync().await.is_err()); assert!(result.is_err()); assert!(!storage.exists(Path::new("/partition/.install-backup")).await.unwrap()); ``` The original writer observes the injected error, but `begin_with_storage` currently returns success and publishes `.install-backup` because `link_tree` synchronizes a handle opened after the error. Proposed regression test name and command: ```bash cargo test -p simulator --lib \ given_a_failed_writeback_when_beginning_an_install_backup_should_refuse_publication \ -- --ignored ``` ### Contribution - [x] I'm willing to submit a pull request to fix this bug -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
