Hi all,

I'd like to propose PIP-493, which makes negative acknowledgement
usable on Key_Shared subscriptions without breaking key ordering.

PIP-379 defined the Key_Shared ordering contract as an invariant, and
then carved negative acknowledgement out of it: messages scheduled for
redelivery by negativecknowledge "will get redelivered as soon as
possible", with "no ordering guarantee in these cases". PIP-493
replaces that carve-out with a guarantee.

The gap matters because negativeAcknowledge is the natural tool for
"try this again in a moment" after a transient downstream failure --
an HTTP 503, a database failover, a rate limit -- and on a Key_Shared
subscription it silently reorders the key. Later messages for that key
are already in the consumer's receiver queue and get processed while
the nacked one waits out its retry delay. The alternatives are worse:
a dead letter topic leaves a permanent gap in the key's sequence, and
blocking the consumer thread stalls every other key that consumer
handles. An application cannot fix this on its own, because only the
broker can stop dispatching messages for a key.

The design has two halves, because at the moment of the nack the
messages that must not reach the application are in two places.

On the broker, a new CommandNegativeAck is sent at nack time and marks
the message's sticky key hash "retrying". While a hash is retrying,
only the earliest negatively acknowledged position of that hash may be
dispatched -- that position is the retry itself -- and the rule is
applied at all three existing dispatch gates. The command carries no
delay: the retry timer, RedeliveryBackoff and the acknowledgement
grouping window all stay on the client exactly as they are today, so
the broker gains no scheduler and the nacked message stays inside the
existing unacknowledged-message accounting.

On the client, messages for the guarded key that are already inside
the consumer are returned to the broker rather than held, and their
redelivery count is not incremented because the application never saw
them. Holding them instead would leave them in that consumer's pending
acknowledgements, so a hash reassignment would make PIP-379 mark the
hash draining until they are acknowledged, while the client refuses to
hand them over until the nacked message is acknowledged -- a circular
wait broken only by an acknowledgement timeout, which is disabled by
default.

The client API additions are opt-in. OrderedNegativeAckMode lets an
application that depends on the guarantee fail fast against a broker
that does not advertise support, while the default preserves today's
silent fallback. Consumer#acquireForOrderedProcessing serves
applications that keep several messages in flight behind per-key
queues, so the library rather than the application decides whether a
queued message may still be processed. A client-side broad-failure
delivery pause answers the common case where the downstream is failing
for every key rather than for one, by stopping flow permits instead of
opening as many concurrent retry cycles as there are active keys.

Nothing is persisted, no metadata or serialized format changes, and
the protocol change is additive. Old clients are unaffected; new
clients against an old broker fall back to today's behaviour.

The full proposal can be found at: https://github.com/apache/pulsar/pull/26418
Rendered PIP document:
https://github.com/lhotari/pulsar/blob/lh-pip-493-keyshared-nack-ordering/pip/pip-493.md

I welcome your feedback and discussion on this proposal. Please share
your thoughts, concerns, or suggestions.

-Lari

Reply via email to