[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037293#comment-17037293
]
ASF GitHub Bot commented on KAFKA-9472:
---------------------------------------
C0urante commented on pull request #8118: KAFKA-9472: Remove deleted tasks from
status store
URL: https://github.com/apache/kafka/pull/8118
[Jira](https://issues.apache.org/jira/browse/KAFKA-9472)
Although the statuses for tasks are removed from the status store when their
_connector_ is deleted, their status are not removed when only the task is
deleted, which happens in the case that the number of tasks for a connector is
reduced.
These changes add logic for deleting the statuses for those tasks from the
status store whenever a rebalance has completed and the leader of a distributed
cluster has detected that there are recently-deleted tasks. Standalone is also
updated to accomplish this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
