diegomrsantos commented on PR #4130: URL: https://github.com/apache/iggy/pull/4130#issuecomment-5779531855
@hubcio @spetz, working on [#4130](https://github.com/apache/iggy/pull/4130) raised a broader question about the guarantees we want from purge. Today, purge clears messages, restarts their offsets at zero, and separately removes consumer bookmarks. If a bookmark deletion is not durable, an old bookmark at `2` can return after power loss and silently skip new messages `0–2`. It still looks valid because those offsets now exist in a different history. Retrying the entire purge can also delete fresh messages accepted after the original reset. That is why this PR separates message reset from bookmark cleanup and records their progress independently. **The fundamental requirement is that messages, bookmarks, and recovery metadata agree about which history they describe.** Physical file deletion can take several steps, provided recovery restores a coherent logical state. A persisted cutoff is only meaningful if the sequence it describes is also recovered consistently. There is a related defect in the current PR. A local partition regression with three replicas and both durability policies set to `Replicated` restores the old operation cutoff, but reopens no durable operation journal. When operations restart at `1`, fresh committed messages at or below the old cutoff disappear from polls. These are internal operation numbers, separate from public message offsets. Existing tests miss this combination because they either restore the previous operation counter or exercise restart with a singleton. I looked at how other systems handle this. - **Kafka** advances the earliest available offset while preserving message numbering. After deleting messages through `100`, new messages continue from `101`. A bookmark at `80` is recognizably outside the available range, and the consumer can resume at the beginning, jump to the end, or report an error according to its configuration. Consumer offsets are managed separately, so message deletion does not require atomically rewriting every bookmark. [Deletion implementation](https://github.com/apache/kafka/blob/4.3.1/core/src/main/scala/kafka/cluster/Partition.scala#L1514-L1539), [consumer behavior](https://kafka.apache.org/43/configuration/consumer-configs/#auto.offset.reset) - **NATS JetStream** preserves numbering even when a stream becomes empty and records the exact boundary of a clustered purge. Recovery replays “purge through 100,” so a later message `101` survives. It adjusts consumer state separately. File cleanup uses prepared replacement storage and recovery logic that can finish an interrupted switch, with old files removed afterward. [Replay protection](https://github.com/nats-io/nats-server/blob/v2.15.0/server/jetstream_cluster.go#L5397-L5443), [storage recovery](https://github.com/nats-io/nats-server/blob/v2.15.0/server/filestore.go#L10619-L10804) - **Redpanda** replicates a command advancing the partition's starting offset, requests a disk flush, and waits for the command to be applied. The committed boundary determines which messages remain available, while background cleanup reclaims obsolete storage. This gives recovery an authoritative deletion decision without requiring all files to disappear together. [Implementation](https://github.com/redpanda-data/redpanda/blob/v26.2.2/src/v/cluster/log_eviction_stm.cc#L240-L304) - **Pulsar** identifies positions by ledger and entry, like a volume number and page number. Entry zero in a new ledger cannot be mistaken for entry zero in the old one. Truncation advances subscription bookmarks before trimming unused ledgers. This is not one transaction across all bookmarks, but it preserves the identity of their positions. [Position model](https://bookkeeper.apache.org/docs/getting-started/concepts/), [truncate implementation](https://github.com/apache/pulsar/blob/v4.0.13/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L4772) - **RabbitMQ quorum queues** order purge with message operations in their replicated history. Replaying “receive A, purge, receive B” leaves B intact because purge retains its place in that history. Physical compaction happens separately. Its semantics differ from Iggy's, since consumer state and messages already delivered but awaiting acknowledgment survive purge. [Purge submission](https://github.com/rabbitmq/rabbitmq-server/blob/v4.3.6/deps/rabbit/src/rabbit_fifo_client.erl#L527), [queue state transition](https://github.com/rabbitmq/rabbitmq-server/blob/v4.3.6/deps/rabbit/src/rabbit_fifo.erl#L501) These systems have different atomicity and durability guarantees. The common ideas worth considering are stable identities, precise replay boundaries, and separating the committed logical change from storage reclamation. I'd appreciate your input on three points. 1. Is restarting public message offsets at zero after purge a required compatibility guarantee, or could we preserve numbering and advance the available boundary? 2. If we keep the reset semantics, would a committed generation shared by message and bookmark validity give us a clearer recovery contract? 3. What should a successful purge guarantee after restart or power loss under each durability policy, and is that guarantee per partition? My preference is to fix the concrete recovery defect in this PR and add the missing restart regression, then evaluate the broader design separately. The tests should also cover failed directory synchronization, retries after fresh writes, and another crash during recovery. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
