skaushik15 opened a new pull request, #18750: URL: https://github.com/apache/iceberg/pull/18750
Currently Iceberg Kafka Connector stalls when it receives bad records: a record that cannot be converted to the table schema fails the task in `put()`, and after every restart the task reads the same record and fails again. This PR implements support for bad record handling as per the configurations. The user will be able to skip bad records instead of stalling, log the bad records if required and publish bad records to the DLQ topic. The DLQ support is implemented using [org.apache.kafka.connect.sink.ErrantRecordReporter](https://kafka.apache.org/26/javadoc/org/apache/kafka/connect/sink/ErrantRecordReporter.html) provided by the Kafka Connect framework itself. The framework applies the following configurations, so the connector does not define them again: - [errors.tolerance](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.tolerance): 'none' (default) reports the record and fails the task; 'all' reports the record and skips it. - [errors.deadletterqueue.topic.name](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.deadletterqueue.topic.name): the DLQ topic for bad records. - [errors.log.enable](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.log.enable) and [errors.log.include.messages](https://kafka.apache.org/32/generated/sink_connector_config.html#sinkconnectorconfigs_errors.log.include.messages): log the bad records. Without a DLQ or error logging the framework provides no reporter, and the task fails as before. Only conversion failures are reported. File, catalog and schema update failures still fail the task, so a transient outage never sends good records to the DLQ. Offsets of reported records are committed with the next Iceberg commit only after their reports have completed. This builds on #14618 by @devendra-nr (credited as co-author), rebased on the current `main`. ## Query engine Kafka Connect Closes #16876 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
