GitHub user lhotari edited a comment on the discussion: Past Acknowledged messages are redelivered
More details that could cause unexpected behavior with the above configuration: The `managedLedgerPersistIndividualAckAsLongArray=true` format's storage size is dependent on the backlog size and not on the number of unacked ranges. A backlog of 30M entries consumes about 5MB of storage (which exceeds the default `nettyMaxFrameSizeBytes`). There's a detail in the current implementation that compression isn't used when the cursor info is stored to bookkeeper. The compression is only used for metadata storage. The problem with the `managedLedgerPersistIndividualAckAsLongArray` format seems to be that after the state cannot be stored, the managed ledger will get into a bad state when the ledger rollover happens. Another issue that is caused when the state exceeds `nettyMaxFrameSizeBytes` is that updates will be first attempted to BK via managed ledger and then retried to metadata store. Since `LZ4` compression handles the metadata storage part, the metadata storage can already store a large amount of individual acks. Storing might succeeed, but the problem is that there will be frequent failures in the large backlog case. A workaround could be to increase `managedLedgerMaxUnackedRangesToPersistInMetadataStore` to a higher value to avoid unnecessary noisy logs, but that's a broker wide config, so it wouldn't be possible to target a single topic / managed ledger where cursors have a large backlog. `managedLedgerDefaultMarkDeleteRateLimit` is the setting to control how frequently the state is stored to the ledger. By default it's once every second. GitHub link: https://github.com/apache/pulsar/discussions/23990#discussioncomment-17242413 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
