diegomrsantos opened a new issue, #4128: URL: https://github.com/apache/iggy/issues/4128
### Bug description Purge can durably record its applied generation even when synchronizing a consumer offset directory fails. If a subsequent power loss loses the offset file deletions but preserves the purge generation, restart can restore the old consumer offsets without retrying the purge. Consumers can then skip messages written after the purge. This affects both individual consumers and consumer groups. The failure path remains after #4092, including when consumer offset durability is `persisted`. This report is based on code inspection. I have not injected the synchronization failure or reproduced the power loss sequence. ### Affected component and versions Iggy server. Inspected `master` at `e0027506f2a0379ee3bb5508427ff570e3a9a435` and #4119 at `d89b81f0297cbcb6577a5a69cd44857c94b0cdf0`. The links below refer to the inspected `master` commit. ### Evidence 1. After deleting consumer offset files, `purge()` synchronizes both offset directories. A failure [only logs a warning](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7548-L7574). It does not return an error or preserve a pending cleanup obligation. 2. Execution can then [persist `purge.gen` and advance `applied_purge_generation`](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7627-L7654). Recording that generation synchronizes its own parent directory, which does not establish durability for deletions inside the separate consumer offset directories. 3. On restart, recovery [loads consumer and group offset files and seeds their values into partition state](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_helpers.rs#L181-L257). It can clamp a value to the recovered offset range, but does not discard it based on the purge generation. A clamped offset is still a consumer entry; "consumed through 0" is different from having no committed offset. 4. The reconciler [only requests another purge when the committed generation exceeds the applied generation](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_reconciler.rs#L1430-L1453). A surviving `purge.gen` therefore prevents the cleanup from being retried for that generation. The durable journal marker added in #4092 does not close this gap. It is [written before offset cleanup](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7336-L7345). Its synchronization cannot cover deletions that occur later. Recovery [restores the journal purge floor](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L829-L832), but that floor does not invalidate the restored consumer offset files. ### Failure sequence derived from the code 1. Persist consumer and group offsets for an existing message history. 2. Purge the topic. Offset file deletions succeed, but synchronization of either offset directory fails. 3. Allow the subsequent write of `purge.gen` to succeed. 4. Lose power before the unsynchronized directory changes become durable. Model the old offset files surviving while the new purge generation survives too. 5. Restart. Recovery loads the old offsets, while the reconciler sees the purge generation as already applied. 6. Poll fresh messages with `Next`. Restored consumer progress can skip messages in the new history. A process restart alone is not a reliable reproduction because the operating system may still retain and flush the pending directory changes. ### Expected behavior and suggested regression The partition must not record the purge as locally applied while removal of its consumer offset files remains unconfirmed on durable storage. A synchronization failure must preserve a retry or recovery obligation. Recovery or retry must also avoid deleting fresh writes that the system has already acknowledged. A deterministic regression should fail synchronization of each offset directory separately after successful deletion. It should verify that purge cannot reach an applied state that suppresses cleanup, then simulate recovery with the old offset files restored. After cleanup completes, both consumer offset maps should be empty and `Next` should return the complete fresh history. Cover both durability policies, a successful synchronization control, and preservation of any acknowledged fresh writes across cleanup retries. ### Related issues - #3793 tracks old operations being replayed through journal repair after purge. This report concerns the durability of local file deletions and does not require repair from another replica. - #4117, addressed by #4119, concerns stale poll completions changing progress after purge. This report does not require a pending poll. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
