raghav-reglobe commented on issue #16876: URL: https://github.com/apache/iceberg/issues/16876#issuecomment-5226814596
+1 on this — adding some production evidence for why the current behavior is more painful than a single task failure, in case it helps prioritize. We run the sink across five Connect clusters on a busy CDC estate (thousands of auto-created tables). The failure mode we keep hitting is not just "task fails on a bad record": on restart/rebalance the task re-reads the same offset, hits the same record, and dies again — a permanent crash-loop pinned on one poison record, while consumer lag for every topic assigned to that task grows unbounded. Real triggers we have hit inside `put()`, all invisible to `errors.tolerance=all` + DLQ: - a required nested struct field the record can no longer fill after a source schema change (NPE in the writer path — this one cost us a multi-day ingest outage for a whole source before we diagnosed it) - type drift between the record and the table (`Cannot convert to boolean: java.lang.Short`, `Cannot convert to string: java.util.Date`) - a malformed payload (`NumberFormatException`) — one record froze a sink's committed offsets for a week while the topic head grew by tens of thousands Recovery today requires operational surgery: either an external "filter copier" (re-append every good record after the poison to the topic head, then move the group offsets past it) or hand-repairing the destination schema. On design: Connect has the errant-record reporter API (`SinkTaskContext.errantRecordReporter()`) exactly for sink-internal failures — wrapping the sink's record conversion per-record and routing failures through it would honor the operator's configured tolerance/DLQ with context headers, without the sink inventing its own error-handling config surface. Happy to test a patch against our deployment, and we have a reliable reproduction for the NPE case if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
