[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037293#comment-17037293
]
ASF GitHub Bot commented on KAFKA-9472:
---
C0urante commented on pull request #8118: KAFKA-9472: Remove deleted tasks from
status store
URL: https://github.com/apache/kafka/pull/8118
[Jira](https://issues.apache.org/jira/browse/KAFKA-9472)
Although the statuses for tasks are removed from the status store when their
_connector_ is deleted, their status are not removed when only the task is
deleted, which happens in the case that the number of tasks for a connector is
reduced.
These changes add logic for deleting the statuses for those tasks from the
status store whenever a rebalance has completed and the leader of a distributed
cluster has detected that there are recently-deleted tasks. Standalone is also
updated to accomplish this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023325#comment-17023325
]
Konstantine Karantasis commented on KAFKA-9472:
---
Sounds good. Thanks for taking a closer look
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023297#comment-17023297
]
Chris Egerton commented on KAFKA-9472:
--
After looking into this a bit I suspect the root cause is different... it looks
like KAFKA-8869 is caused by not removing task configurations for deleted
connectors from the task config map, whereas this issue is caused by not
removing the statuses for no-longer-needed tasks from the status backing store
when the number of tasks for a connector is reduced, but the connector is still
running.
>From what I can tell, this issue does not arise when a connector is deleted
>(the task statuses are correctly removed from the backing store), and
>KAFKA-8869 does not arise when the number of tasks for a connector is reduced
>(the task reduction is correctly reflected in the values for the task config
>map).
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023280#comment-17023280
]
Konstantine Karantasis commented on KAFKA-9472:
---
I overlooked the backing stores. But, yes, I'm suspecting a common root cause,
which has to do with the fact that we don't remove the task from the config
backing store, which would be reflected to the status backing store. More
investigation might be granted, that's why I didn't close the ticket. But I
wanted to bring up that this sounded relevant to KAFKA-8869.
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023271#comment-17023271
]
Chris Egerton commented on KAFKA-9472:
--
[~kkonstantine] the issues may be similar but I don't see how this is a
duplicate. KAFKA-8869 appears to be related to the config backing store whereas
this issue is related to the status backing store. Do you suspect a shared root
cause that is not explicitly mentioned in either ticket?
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9472) Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
[
https://issues.apache.org/jira/browse/KAFKA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023264#comment-17023264
]
Konstantine Karantasis commented on KAFKA-9472:
---
Thanks for reporting [~ChrisEgerton]
I’m suspecting this ticket might be a duplicate of:
https://issues.apache.org/jira/browse/KAFKA-8869
> Reducing number of tasks for connector causes deleted tasks to show as
> UNASSIGNED
> -
>
> Key: KAFKA-9472
> URL: https://issues.apache.org/jira/browse/KAFKA-9472
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2,
> 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Priority: Major
>
> If a connector is successfully created with {{t1}} running tasks and then
> reconfigured to use {{t1 - n}} tasks (where {{t1}} and {{n}} are both whole
> numbers and {{n}} is strictly less than {{t1}}), the connector should then
> list {{t1 - n}} total tasks in its status (which can be queried via the
> {{/connectors/:name:/status}} endpoint or the {{/connectors}} endpoint with
> the {{expand}} URL query parameter set to {{status}}).
> However, the connector will instead continue to list {{t1}} total tasks in
> its status, with {{n}} of them being listed as {{UNASSIGNED}} and the
> remaining {{t1 - n}} of them being listed as {{STARTED}}.
> This is because the only time a task status is removed from the status
> backing store (as opposed to simply being updated to {{UNASSIGNED}}) is when
> its connector is deleted. See relevant code snippets from the
> [AbstractHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L187-L192]
> and
> [DistributedHerder|https://github.com/apache/kafka/blob/df13fc93d0aebfe0ecc40dd4af3c5fb19b35f710/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1511-L1520]
> classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
