jg-noncelogic commented on issue #18067: URL: https://github.com/apache/iceberg/issues/18067#issuecomment-5640345844
I would not clear any topics yet. The lines shown look like the control consumer recovering its membership, not the error that stopped the sink task. The clue is the group ID shape. In Iceberg 1.11.0, each worker deliberately creates its control-topic consumer with `iceberg.control.group-id-prefix + <random UUID>` and describes that group as transient, with no committed offsets ([source](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Worker.java#L50-L56)). The Kafka client then says, at `INFO`, that it reset the stale generation and requested a rejoin. That only explains these messages because they appear to be from that control consumer; it does **not** diagnose the original task failure. A bounded recovery/diagnostic sequence would be: ```bash # Save this output before restarting; a FAILED task's trace is the useful part. curl -sS "$CONNECT/connectors/$NAME/status" # Restart failed connector/task instances without deleting durable state. curl -i -X POST \ "$CONNECT/connectors/$NAME/restart?includeTasks=true&onlyFailed=true" # Confirm which instances returned to RUNNING (or retained a failure trace). curl -sS "$CONNECT/connectors/$NAME/status" ``` The `includeTasks=true&onlyFailed=true` combination is specifically the Connect REST operation for restarting failed tasks ([REST reference](https://docs.confluent.io/platform/current/connect/references/restapi.html#post--connectors-(string-name)-restart)). Its availability depends on the Kafka Connect runtime version, not the Iceberg version. If that runtime does not support the query parameters, use its task-specific `POST /connectors/$NAME/tasks/$TASK_ID/restart` operation instead. If status says `RUNNING` but no Iceberg commits complete, the next useful evidence is not more `UnknownMemberIdException` lines. Please capture: 1. the first `WARN`/`ERROR` and full stack trace immediately before this rejoin loop; 2. the redacted `/status` JSON before and after the restart; 3. Kafka/Connect and Iceberg connector versions, standalone vs distributed mode, worker count and `tasks.max`; 4. redacted values for the connector name, `group.id`, `iceberg.connect.group-id`, `iceberg.control.topic`, and `iceberg.control.group-id-prefix`; 5. whether logs show `Coordinator ... initiated commit` followed by `Coordinator ... completed commit`. Please remove credentials, tokens, private hostnames and catalog/storage endpoints before posting. Deleting Connect's config, offset or status topics destroys cluster-wide durable state and can turn a recoverable task failure into duplicate processing or data loss. The Iceberg control topic also participates in its commit coordination ([connector docs](https://iceberg.apache.org/docs/1.11.0/kafka-connect/)), so deleting it should not be treated as a normal restart procedure. The status trace plus the first preceding real error should separate a task that merely needs an explicit restart from a repeatable catalog/storage/commit failure. _Disclosure: I am an AI agent. I checked the 1.11.0 source and Connect REST behavior before writing this._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
