[jira] [Commented] (KAFKA-15309) Add custom error handler to Producer
[ https://issues.apache.org/jira/browse/KAFKA-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761924#comment-17761924 ] Matthias J. Sax commented on KAFKA-15309: - Sure, the ticket is up for grabs. Note, that we will need a KIP for this to get a proper and approved design -> [https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals] Let us know if you have any questions about the KIP process. > Add custom error handler to Producer > > > Key: KAFKA-15309 > URL: https://issues.apache.org/jira/browse/KAFKA-15309 > Project: Kafka > Issue Type: New Feature > Components: producer >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > Attachments: KafkaProducerReproducer.java, app.log > > > The producer collects multiple records into batches, and a single record > specific error might fail the whole batch (eg, `RecordTooLargeException`). > This ticket suggests to add a per-record error handler, that allows user to > opt into skipping bad records without failing the whole batch (similar to > Kafka Streams `ProductionExceptionHandler`). > The fix of https://issues.apache.org/jira/browse/KAFKA-9279 caused > https://issues.apache.org/jira/browse/KAFKA-15259 which inspired this ticket. > Another example for which a production exception handler could be useful, if > a user tries to write into a non-existing topic, which returns a retryable > error code; with infinite retries the producer would hang retrying forever. A > handler could help to break the infinite retry loop. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15309) Add custom error handler to Producer
[ https://issues.apache.org/jira/browse/KAFKA-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757189#comment-17757189 ] Fei Xie commented on KAFKA-15309: - Hi [~mjsax] , I am new to Kafka (I used Kafka daily in my work but new to dev). From the ticket description, this seems to be a fairly standalone new feature with reproducible steps. Can I pick up this one? > Add custom error handler to Producer > > > Key: KAFKA-15309 > URL: https://issues.apache.org/jira/browse/KAFKA-15309 > Project: Kafka > Issue Type: New Feature > Components: producer >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > Attachments: KafkaProducerReproducer.java, app.log > > > The producer collects multiple records into batches, and a single record > specific error might fail the whole batch (eg, `RecordTooLargeException`). > This ticket suggests to add a per-record error handler, that allows user to > opt into skipping bad records without failing the whole batch (similar to > Kafka Streams `ProductionExceptionHandler`). > The fix of https://issues.apache.org/jira/browse/KAFKA-9279 caused > https://issues.apache.org/jira/browse/KAFKA-15259 which inspired this ticket. > Another example for which a production exception handler could be useful, if > a user tries to write into a non-existing topic, which returns a retryable > error code; with infinite retries the producer would hang retrying forever. A > handler could help to break the infinite retry loop. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15309) Add custom error handler to Producer
[ https://issues.apache.org/jira/browse/KAFKA-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752369#comment-17752369 ] Tomonari Yamashita commented on KAFKA-15309: FYI, I've created a simple reproducer. The Kafka Producer behavior has changed since Kafka 3.2.0 implemented KAFKA-9279, which forces the transaction to abort if any of the individual producer.send()'s fail, and this change is expected behavior. This reproducer is able to confirm that the Kafka Producer behavior has changed. In addition, maybe, by integrating the extension of the error handler provided in this ticket, it could be used as the basis for a reference implementation that skips wrong messages. [Reproduction procedure] # Create "input-topic" topic and "output-topic" # Put several messages on "input-topic" # Execute a simple Kafka Producer program that transfers too large messages from "input-topic" to "output-topic" with transaction and ignores RecordTooLargeException of producer.send(). Please refer to the reproducer program (attached file: KafkaProducerReproducer.java). # ==> Changing the version of Kafka Producer will lead to a different behavior whether commit is available or not (i.e., the transaction aborts), as follows: -- Kafka 2.8.2 : It is possible to commit ignoring RecordTooLargeException -- Kafka 3.0.0 : It is possible to commit ignoring RecordTooLargeException -- Kafka 3.1.2 : It is possible to commit ignoring RecordTooLargeException -- Kafka 3.2.0 : Cannot skip messages by ignoring RecordTooLargeException because the transaction was rollbacked/aborted. -- Kafka 3.2.3 : Cannot skip messages by ignoring RecordTooLargeException because the transaction was rollbacked/aborted. -- Kafka 3.5.1 : Cannot skip messages by ignoring RecordTooLargeException because the transaction was rollbacked/aborted. > Add custom error handler to Producer > > > Key: KAFKA-15309 > URL: https://issues.apache.org/jira/browse/KAFKA-15309 > Project: Kafka > Issue Type: New Feature > Components: producer >Reporter: Matthias J. Sax >Priority: Major > Labels: needs-kip > Attachments: KafkaProducerReproducer.java, app.log > > > The producer collects multiple records into batches, and a single record > specific error might fail the whole batch (eg, `RecordTooLargeException`). > This ticket suggests to add a per-record error handler, that allows user to > opt into skipping bad records without failing the whole batch (similar to > Kafka Streams `ProductionExceptionHandler`). > The fix of https://issues.apache.org/jira/browse/KAFKA-9279 caused > https://issues.apache.org/jira/browse/KAFKA-15259 which inspired this ticket. > Another example for which a production exception handler could be useful, if > a user tries to write into a non-existing topic, which returns a retryable > error code; with infinite retries the producer would hang retrying forever. A > handler could help to break the infinite retry loop. -- This message was sent by Atlassian Jira (v8.20.10#820010)
