[
https://issues.apache.org/jira/browse/KAFKA-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072323#comment-17072323
]
ASF GitHub Bot commented on KAFKA-9740:
---------------------------------------
lizihan021 commented on pull request #8401: KAFKA-9740 Add a continue option
for Kafka Connect error handling
URL: https://github.com/apache/kafka/pull/8401
### Pull Request for KIP-582
Some background: currently there are two error handling options in Kafka
Connect, "none" and "all". Option "none" will config the connector to fail
fast, and option "all" will ignore broken records.
If users want to store their broken records, they have to config a broken
record queue, which is too much work for them in some cases.
Some sink connectors have the ability to deal with broken records, for
example, a JDBC sink connector can store the broken raw bytes into a separate
table, a S3 connector can store that in a zipped file.
Therefore, it would be ideal if Kafka Connect provides an additional option
that sends the broken raw bytes to SinkTask directly.
Wiki:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148642653
JIRA: https://issues.apache.org/jira/browse/KAFKA-9740
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add a "continue" option for Kafka Connect error handling
> --------------------------------------------------------
>
> Key: KAFKA-9740
> URL: https://issues.apache.org/jira/browse/KAFKA-9740
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Zihan Li
> Priority: Major
> Attachments: callback.php, con fig.php, cron_follow.php,
> cron_tweet.php, e_ce.php, e_cji.php, e_maf.php, e_ml.php, e_rl.php,
> f_afs.php, f_fe.php, f_stf.php, favicon.ico, google01293ecf4842b4fe.html,
> google3b7597efe3aa852a.html, index.php, install_tables.php, l_ls.php,
> l_vfl.php, l_vtl.php, license.txt, log_in.php, redirect.php, t_pqt.php,
> t_rt.php, t_st.php
>
>
> Currently there are two error handling options in Kafka Connect, "none" and
> "all". Option "none" will config the connector to fail fast, and option "all"
> will ignore broken records.
> If users want to store their broken records, they have to config a broken
> record queue, which is too much work for them in some cases.
> Some sink connectors have the ability to deal with broken records, for
> example, a JDBC sink connector can store the broken raw bytes into a separate
> table, a S3 connector can store that in a zipped file.
> Therefore, it would be ideal if Kafka Connect provides an additional option
> that sends the broken raw bytes to SinkTask directly.
> SinkTask is then responsible for handling the unparsed bytes input.
> The benefits of having this additional option are:
> * Being user friendly. Connectors can handle broken record and hide that
> from clients.
> * Providing more flexibility to SinkTask in terms of broken record handling.
> Wiki page: [https://cwiki.apache.org/confluence/x/XRvcC]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
