Thanks for the feedback!
To answer your questions:
In general, yes, user topics should exist, however, we support "dynamic
routing" allowing users to extract an output topic name dynamically from
the message content. A poison pill record could contain an invalid topic
a name, and thus, it think it's useful to allow users to handle this
case gracefully, by dropping such message on the floor.
I think that (not) writing into an output topic and dropping a result
records, is of a different quality than (not) writing into a changelog
topic (for which state store and changelog would diverge and we would
pollute state...) -- regular repartitions topics are a little bit
in-between but FK-join subscription/response topics are again more on
the side of changelogs IMHO with regard of impact if we drop a write...
But it was meant as a consideration to be more strict with internal
topics. Happy to just drop the sentence from the KIP -- in the end,
users can check the topic name which is passed into the handler and
exclude internal topic themselves and always FAIL for them.
For processing exception handler, I would rather alter KIP-1033 which is
not shipped yet.
For deserialization exceptions: while Kafka does not ship with schema
registry, we have seen this issue in practice that RETRY could be
useful. However, the question is if the deserializer should just retry
internally? While I am overall not against it, it seems to be related
but orthogonal, and I would like to keep the scope of this KIP focused.
Maybe just file a ticket for it so we can track it.
-Matthias
On 7/2/24 6:50 AM, jiang dou wrote:
Thanks for the KIP
I think adding ProductionExceptionHandlerResponse.RETRY is very valuable
But I agree with Sophie Blee-Goldman: all topics in the same way.
The caller can determine whether the topic exists and take appropriate
action.
The judgment logic of the ProductionExceptionHandler will also be simpler.
Sophie Blee-Goldman <sop...@responsive.dev> 于2024年7月2日周二 07:24写道：
Thanks for the KIP -- definitely agree with this proposal, just have a few
suggestions:
1. In the KIP, you mention
We might also consider to not calling the handler when writing into
internal topics, as those must exist.
Personally I would vote to consider all topics the same in this regard, and
don't fully
understand why we would treat internal topics differently. Presumably, both
internal
and user topics "must exist" by the time we attempt to write anything into
them, no?
Can you maybe clarify this part a bit further?
2.
Although I understand the stated purpose of this KIP is specifically around
the
ProductionExceptionHandler and the case of missing topics, a RETRY option
seems
like it would be useful in general and could also make sense for the new
ProcessingExceptionHandler. WDYT about adding RETRY to the processing
exception
handler as well?
I guess that also begs the question, what about the
DeserializationExceptionHandler?
Most serdes are probably (hopefully!) deterministic, but could
deserialization fail due
to network errors or other transient issues with schema registry?
On Sun, Jun 30, 2024 at 12:35 PM Matthias J. Sax <mj...@apache.org> wrote:
Hi,
as a follow up to https://issues.apache.org/jira/browse/KAFKA-16508
which is related to KIP-1038, I would like to prose adding a RETRY
option to production error handler responses:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627309
Looking forward to your feedback.
-Matthias
