C0urante commented on code in PR #12490:
URL: https://github.com/apache/kafka/pull/12490#discussion_r940338098
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java:
##########
@@ -853,6 +853,9 @@ private void processConnectorConfigRecord(String
connectorName, SchemaAndValue v
connectorConfigs.remove(connectorName);
connectorTaskCounts.remove(connectorName);
taskConfigs.keySet().removeIf(taskId ->
taskId.connector().equals(connectorName));
+ deferredTaskUpdates.remove(connectorName);
+ connectorTaskCountRecords.remove(connectorName);
Review Comment:
Unfortunately, we can't safely remove this entry. There may still be zombie
tasks running for this connector that would need to be fenced out before
bringing up new tasks if it got recreated.
One of the [rejected alternatives in
KIP-618](https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Cleanupoftaskcountrecordsonconnectordeletion)
was to perform a final round of fencing for source connectors after they were
deleted. The rationale was that the work would be unnecessary if the connector
were never recreated, and would provide little advantage if it were (since the
total number of required fencings would remain unchanged).
The point that the these task count records can grow monotonically is an
interesting one, but I'm not sure it's enough to tip the balance towards a
fence-on-delete approach. We don't wipe offsets for deleted source connectors,
and although this has been a pain point for many users who would like to reset
those offsets without having to rename their connectors, to my knowledge, the
storage cost for these offsets has not been an issue.
Have you observed practical fallout from this issue, or is this more of a
general exercise in code cleanliness?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
