yashmayya commented on code in PR #12490:
URL: https://github.com/apache/kafka/pull/12490#discussion_r940928631
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java:
##########
@@ -853,6 +853,9 @@ private void processConnectorConfigRecord(String
connectorName, SchemaAndValue v
connectorConfigs.remove(connectorName);
connectorTaskCounts.remove(connectorName);
taskConfigs.keySet().removeIf(taskId ->
taskId.connector().equals(connectorName));
+ deferredTaskUpdates.remove(connectorName);
+ connectorTaskCountRecords.remove(connectorName);
Review Comment:
Tbh, it doesn't really seem like it's worth the mess of null handling
everywhere. I'm gonna back out this change and make this a single line PR 😆
Btw, unrelated to this PR - but I noticed that the task config generations
aren't persisted to the config topic, they're simply maintained in the
in-memory map. So, based on worker restarts, different workers (herders to be
precise) could have different views on the task config generations for the same
connector because the config topic is compacted and older task commit records
could be lost (the task config generation is incremented by 1 each time a task
commit record is encountered). From what I can tell, task config generations
are used in two places primarily -
i) We compare the task config generation from prior to a EOS task startup
with the task config generation after the task has initialized its
transactional producer (and fail it if there's a mismatch indicating a new set
of tasks have been brought up)
ii) At the end of a zombie fencing, if the task config generation is greater
than the task config generation at the beginning of the zombie fencing, a `409`
is returned by the fencing endpoint because a new set of tasks were brought up
in the meanwhile
I'm struggling to think of any cases where this would cause an issue (i.e.
different herders having different values for task config generations of the
same connector) but I was hoping that you could verify my understanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
