Hello Kafka Streams team,
I’m working on implementing centralized error handling and retry logic in
Kafka Streams using the new ProcessorWrapper API in Kafka 4.0. My goal is
to intercept exceptions uniformly across all processors, route failed
records to dedicated error/retry topics, implement circuit breaker logic,
etc.—without duplicating error-handling logic across the topology.
Current Limitation (my understanding)
Today implementing cross-cutting exception handling requires one of the
following:
• Adding try/catch plus DLQ routing logic inside every processor
• Introducing intermediate branches or topics solely to enable error
routing
• Using ProductionExceptionHandler, which is limited to sink-level
production errors and does not support custom routing
With ProcessorWrapper, we can now intercept all processors uniformly, but
we still hit a limitation: context.forward() can only route to downstream
nodes already wired into the DAG immediately after the current node. There
is currently no mechanism to forward a failed record to an “escape hatch”
sink unless that sink is explicitly parented to every processor that might
produce errors.
This makes it impossible to define a single, global DLQ sink that all
processors may forward errors to without injecting additional topology
structure everywhere.
Historical Context
In Kafka Streams 3.x, I built a similar mechanism by generating additional
topology nodes programmatically:
• Every processor automatically branched into success/error paths
• Logic existed to route messages to various topics based on interceptor
output
It worked very well but required significant infrastructure, boilerplate,
and topology-rewriting logic that was hard to reason about and maintain. It
also had the downside of being very susceptible to breaking changes in
future versions as the API evolves.
Kafka 4.0’s ProcessorWrapper is a huge step forward—it gives us a clean,
centralized interception point. I would prefer not to recreate the
topology-rewriting machinery simply to support error/retry routing.
Desired Behavior
What I’d love to see is a global “escape hatch” sink that any processor can
route errors to without altering the main DAG.
Proposed Enhancements
1. Globally addressable “escape hatch” sinks
Introduce sink nodes that can be targeted by name using
context.forward(), even if they are not direct topology children.
Such sinks would:
• Not change the main topology DAG semantics
• Not participate in normal routing
• Serve as universal destinations for exception, retry, or audit flows
(etc..)
2. Enhanced ProcessingExceptionHandler behavior
Extend it to support:
• Constructing alternate error/retry records
• Sending to user-specified topics
• Allowing normal stream processing to continue after an error is
handled
Either or both of these would provide a very powerful addition to the
Streams library.
Use Cases Enabled
• Centralized DLQ and retry logic
• Global exception auditing
• Circuit breakers
• Early termination flows
• Cross-cutting observability and policy enforcement
My previous implementation was very powerful and made handling these
concerns trivially easy. It sped up the development of our apps
significantly and made understanding the semantics across the entire system
much easier. Using ProcessorWrapper or ProcessingExceptionHandler would
reduce the complexity massively, as well as making implementations much
more future-proof.
If this sounds reasonable, I'd love to help any way I can to make it happen.
Best,
Dave Cracauer
--
This email and any attachments may be confidential or proprietary. Any
review, use, disclosure, distribution or copying of this email is
prohibited except by or on behalf of the intended recipient. If you
received this message in error or are not the intended recipient, please
delete or destroy the email message and any attachments or copies and
notify the sender of the erroneous delivery by return email. To the
extent
that this message or its attachments were sent without
encryption, we
cannot guarantee that the contents have not been changed
or tampered with.
Any advice expressed in this message is being
delivered to you solely for
your use in connection with the matters
addressed herein and may not be
used for any other purpose without our
prior written consent.
