[
https://issues.apache.org/jira/browse/KAFKA-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864879#comment-17864879
]
Matthias J. Sax commented on KAFKA-10356:
-----------------------------------------
Is the description of this ticket still correct? Based on
https://issues.apache.org/jira/browse/KAFKA-16508 which we addressed recently,
we original ended up with an infinite retry loop (at least in newer releases),
and did not call the ProductionExceptionHandler.
The new behavior with K16508 is to call the handler and allow user to drop the
record on the floor. However, I would not consider this as a "lost silently"
case, because the users' custom handler did make this decision.
I don't think that triggering a rebalance and shutting down if a sink topic
does not exist is the right thing any longer – maybe it was a good idea back in
the day, for which we did not have "dynamic routing" feature. But with "dynamic
routing", we should just rely on the production exception handler IMHO, and
allow to drop a potential poison pill record.
Related is https://issues.apache.org/jira/browse/KAFKA-17057 proposing to a
RETRY to the handler.
Any objections to just close this ticket?
> Handle accidental deletion of sink-topics as exceptional failure
> ----------------------------------------------------------------
>
> Key: KAFKA-10356
> URL: https://issues.apache.org/jira/browse/KAFKA-10356
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Bruno Cadonna
> Priority: Major
>
> Today when sink topics are deleted, the producer's send callback would
> eventually return the UnknownTopicOrPartition exception after configured
> max.delivery.ms whose default is 2min if EOS is not enabled (otherwise its
> Integer.MAX_VALUE). Then in Streams implementation the exception would be
> handled by ProductionExceptionHandler which by default would treat it as
> `FAIL` and would cause the thread to die. If it treat it is CONTINUE then it
> would be silently ignored and the sent records are lost silently.
> We should improve this situation in Streams by special-handling
> {{UnknownTopicOrPartition}} exception and trigger a rebalance as well, and
> then in leader we can also check if the sink topic metadata exists, just like
> source topic, and then follow the same logic as in
> https://issues.apache.org/jira/browse/KAFKA-10355.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
