yadavay-amzn commented on code in PR #16434:
URL: https://github.com/apache/iceberg/pull/16434#discussion_r3583502022
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java:
##########
@@ -150,22 +152,57 @@ protected boolean receive(Envelope envelope) {
private void commit(boolean partialCommit) {
try {
doCommit(partialCommit);
+ if (!partialCommit) {
+ consecutiveCommitFailures = 0;
+ }
} catch (RuntimeException e) {
if (partialCommit) {
LOG.warn(
"Partial commit {} failed for task {}, will retry",
commitState.currentCommitId(),
taskId,
e);
- } else {
- LOG.error("Commit {} failed for task {}",
commitState.currentCommitId(), taskId, e);
+ return;
+ }
+
+ if (!isCommitFailedException(e)) {
Review Comment:
You're right - castAndThrow rethrows the bare CommitFailedException (it
extends RuntimeException) in both the single- and multi-thread Tasks paths, so
the cause walk never actually triggered. Simplified it to a direct instanceof
check and changed the test to throw a bare CommitFailedException instead of the
wrapped one. Thanks for the catch.
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java:
##########
@@ -235,6 +239,12 @@ private static ConfigDef newConfigDef() {
120000L,
Importance.LOW,
"config to control coordinator executor keep alive time");
+ configDef.define(
+ COMMIT_MAX_CONSECUTIVE_FAILURES_PROP,
+ ConfigDef.Type.INT,
+ COMMIT_MAX_CONSECUTIVE_FAILURES_DEFAULT,
+ Importance.MEDIUM,
+ "Maximum number of consecutive commit failures before the coordinator
terminates");
Review Comment:
Done - added ConfigDef.Range.atLeast(1) on the config and documented it in
kafka-connect.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
