Hi Uladzislau,
Thanks for the careful review. Answers to your questions below. Happy to hear
thoughts before I update the KIP.
> On the PAPI vs. DSL-only question
Honestly, I'm not fully certain about this myself. I aligned with Matthias's
suggestion in the JIRA discussion to expose writeToDlq() on ProcessingContext,
but I want to revisit this explicitly now that the question is on the table.
The case for PAPI exposure is mainly convenience and parity. PAPI users who
want DLQ behavior today can build it themselves with their own producer, but
they can't reuse what KIP-1034 already wired up: the standard header schema
(__streams.errors.*), the configured DLQ topic, or the internal producer.
Exposing writeToDlq() would let them inherit that infrastructure instead of
maintaining a parallel implementation.
The alternative you mentioned, putting this on InternalProcessorContext so the
DSL gets it without expanding public API, keeps the scope tight but means PAPI
users still maintain their own DLQ plumbing.
I'd appreciate input from you and others on the thread on whether the parity is
worth the broader scope, or whether the internal-only path is the better fit.
> bloku1: Compatibility section contradiction
You're right, that section is wrong as written. Adding a method to
ProcessingContext is a public API change, and I'll fix the section.
For the implementation, I was thinking of defining writeToDlq() as a `default`
method on the interface to preserve compatibility. The default would be a
no-op, and ProcessorContextImpl would override it with the actual DLQ-write
logic. Does that approach sound reasonable, or would you prefer a different one?
> bloku2: Behavior when no DLQ topic is configured
My current thinking is to follow KIP-1034's behavior here: when
ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG is not set, writeToDlq() is a no-op
and the observable behavior matches today's (the record is dropped, the
existing WARN log from the DSL operator fires, and dropped-records-total
increments). I'd document this in the Javadoc. Let me know if you think we
should diverge from KIP-1034 here.
> bloku3: Per-call topic override
Fair point. KIP-1034 already allows custom handlers to write to arbitrary DLQ
topics by constructing ProducerRecord<>(customTopic, ...) themselves. To stay
consistent, one option is to add an overload:
default void writeToDlq(Record<?, ?> record, Exception exception) {
writeToDlq(record, exception, null);
}
default void writeToDlq(Record<?, ?> record, Exception exception, String
dlqTopic) {
// no-op by default; overridden by ProcessorContextImpl
}
The two-argument form uses the configured default topic. The three-argument
form lets callers route specific dropped records to specific topics. If
dlqTopic is null and no default is configured, the call is a no-op as described
in bloku2.
Would this overload work, or do you (or others on the thread) have a different
preference for how to expose per-call routing?
Let me know your thoughts, and I'll update the KIP accordingly.
Thanks,
Daeho
-----Original Message-----
From: "Uladzislau Blok"<[email protected]>
To: <[email protected]>;
Cc:
Sent: 2026-05-02 (토) 21:17:55 (GMT+09:00)
Subject: Re: [DISCUSS] KIP-1328: Dead Letter Queue Support for DSL Dropped
Records
Dear Daeho Kwon,
Thanks for the KIP!
From my understanding of the JIRA ticket, two proposals were evaluated:
- Exposing the DLQ method to PAPI (affects public API).
- Creating an internal utility for DSL operators or extending
InternalProcessorContext instead of the public one (this still affects the
public DLQ API, but avoids changes to PAPI).
If we choose to change PAPI, we likely need to extend the scope of the KIP.
Currently, it focuses on the DSL, but adding this method to
ProcessingContext would enable it for all clients. I suggest revisiting the
proposals to consider the trade-offs between using an internal utility for
the DSL case versus generally enabling this functionality for PAPI.
I also have the following questions regarding the KIP:
- bloku1: The "Compatibility, Deprecation, and Migration Plan" section
states there are no changes to the public API, but adding a method to
ProcessingContext seems to contradict this. Can we clarify this point?
- bloku2: What happens if a user calls the writeToDlq method without
specifying a DLQ topic name via the
ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG property?
- bloku3: KIP-1034 allows users to send records to a specific topic even
when a default DLQ topic is set. Do we plan to allow this in KIP-1328?
Thanks,
Uladzislau Blok
On Mon, Apr 27, 2026 at 1:43 PM 권대호 <[email protected]> wrote:
>
> Hi all,
> I'd like to start a discussion on KIP-1328: Dead Letter Queue Support
for DSL Dropped Records.
>
> KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1328%3A+Dead+Letter+Queue+Support+for+DSL+Dropped+Records
> JIRA: https://issues.apache.org/jira/browse/KAFKA-20439
>
> Kafka Streams DSL operators currently drop certain records silently
(e.g., null key/value, expired window records) with only a log warning and
a metric increment. There is no way to capture these records
> for analysis or reprocessing.
>
> This KIP proposes adding a writeToDlq() method to ProcessingContext,
allowing DSL operators to route dropped records to the existing DLQ
infrastructure introduced in KIP-1034.
>
> Looking forward to your feedback.
>
> Thanks,
> Daeho Kwon
