diegomrsantos opened a new issue, #4117: URL: https://github.com/apache/iggy/issues/4117
## Summary Disk polls run asynchronously so that file I/O does not block the shard that owns a partition. The poll captures the resources it needs, performs the read in a detached task, and completes later. The detached poll currently also has access to shared consumer progress state. When the read finishes, it can apply an automatic commit or update a consumer group's `last_polled` value before the partition owner establishes that the result still belongs to the current message history. This becomes incorrect when a topic is purged while the disk read is pending. Purge removes the old messages, clears consumer progress, and starts a new offset space at zero. The pending poll is not revoked. It can therefore finish with information from the history that existed before the purge and apply that information to the history that exists after the purge. ## Example Consider an individual consumer that polls with `Next` and automatic commit enabled: | Time | Partition state | Delayed poll | | --- | --- | --- | | 1 | Old messages occupy offsets 0 through 2 | A disk poll starts for offsets 0 through 2 | | 2 | The topic is purged; messages and consumer progress are cleared | The disk read is still pending | | 3 | Five new messages are appended at offsets 0 through 4 | The pending poll still belongs to the old history | | 4 | The new history is ready for consumers | The old poll completes and records offset 2 | | 5 | The consumer issues its next `Next` poll | Polling resumes after offset 2 | The consumer can now skip the new messages at offsets 0 through 2. Checking only whether offset 2 is within the current range cannot detect the problem because offset values are reused after purge. The same number identifies different messages before and after the purge. If fewer new messages exist, the stale progress can instead move the consumer beyond the current end of the topic and cause empty polls until the new history catches up. ## Why the current completion checks are insufficient `PollPlan` is created while the partition is borrowed by its owning shard. To let the read continue after that borrow ends, the plan captures owned file resources and shared handles for consumer offsets, automatic commit capacity, durable offset state, and group `last_polled` state. For a disk poll, `PollPlan::execute` runs outside the partition owner. After the file read completes, it can update the captured offset map and produce an `AutoCommitApplied` value. The server later borrows the partition again to decide whether to submit the corresponding `StoreConsumerOffset` operation. If that decision rejects the operation, the earlier local update is rolled back. This protects some changes in partition ownership and consensus state, but it does not prove that the message history is unchanged. A purge clears the existing shared maps rather than replacing every handle captured by older polls. A plan created before the purge can therefore still point at the same allocation after the purge. The allocation is current, but the read result is not. There is a similar path for consumer groups. A poll without automatic commit can update `last_polled` directly from the detached completion, without returning to the partition owner for history validation. This means the current flow mixes two responsibilities: 1. Reading message data, which may wait for file I/O and may safely happen outside the owner. 2. Deciding whether the result is still valid and applying consumer progress, which depends on the partition's current history and must be serialized with changes to that history. The second responsibility cannot be made safe by validating offset numbers alone. ## Observable impact Depending on the poll type and timing, a completion from before the purge can: - advance an individual consumer's offset in the new history; - advance a consumer group's `last_polled` value in the new history; - enqueue or replicate an automatic commit derived from the old history; - cause a later `Next` poll to skip messages that were appended after the purge; - authorize a reply whose message data and current offset were captured from a history that no longer exists. The race requires a disk read to remain pending across purge, so it may be uncommon in normal tests. Slow storage, a busy runtime, or a deliberately suspended read makes the ordering reproducible. ## Expected behavior A poll result must only affect the partition history from which it was created. If that history has been replaced before completion: - the result must be rejected with a retryable error; - no messages from that result should be returned to the client; - consumer offsets and group `last_polled` values must remain unchanged; - no automatic commit or capacity reservation from that result may survive. This rule must still hold when the stale offset is numerically valid in the new history. ## Suggested test A deterministic regression test can expose the bug without relying on timing: 1. Persist messages at offsets 0 through 2 so polling uses the disk path. 2. Start a poll for those messages and suspend it at the file read. 3. Purge the topic while the poll is suspended. 4. Append new messages at offsets 0 through 4 and persist them. 5. Resume the old poll and run its completion path. 6. Verify that the completion is rejected and that consumer progress remains empty. 7. Poll with `Next` and verify that all five new messages are returned, starting at offset 0. The regression should cover an individual consumer with automatic commit and a consumer group whose `last_polled` value would otherwise change. A poll without automatic commit should also be checked to ensure that an obsolete result cannot authorize a reply. ## Possible implementation direction Keep the asynchronous worker limited to reading messages. It should return the read fragments, relevant immutable request context, and an identity for the message history from which the read was planned. It should not hold handles that allow it to change consumer progress. The partition owner can then receive the result, compare its history identity with the current one, perform admission and capacity checks, apply progress, and authorize the reply as one synchronous completion step. Any queued automatic commit should retain the same history identity so it can be discarded if the history changes before operation assignment. The exact representation can be decided in the fix. The required invariant is that work created for an old partition history cannot change progress or authorize a successful poll in a newer history. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
