[
https://issues.apache.org/jira/browse/KAFKA-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860408#comment-17860408
]
yazgoo edited comment on KAFKA-10413 at 6/27/24 1:38 PM:
---------------------------------------------------------
Hello, I launch the attached script :
[^rebalance.sh]
And in my test after waiting for a few minutes, I get:
one connect well balanced
{code:java}
❯ curl -s http://localhost:8081/connectors/s3-connector2/status | jq .tasks
|grep worker_id | sort | uniq -c
15 "worker_id": "k2:8082"
15 "worker_id": "k3:8083"
15 "worker_id": "k4:8084"
15 "worker_id": "k5:8085"
15 "worker_id": "k6:8086"
15 "worker_id": "k7:8087"
15 "worker_id": "k8:8088"
15 "worker_id": "k9:8089"
{code}
And the other one unbalanced
{code:java}
❯ curl -s http://localhost:8081/connectors/s3-connector1/status | jq .tasks
|grep worker_id | sort | uniq -c
27 "worker_id": "k1:8081"
11 "worker_id": "k2:8082"
12 "worker_id": "k3:8083"
11 "worker_id": "k4:8084"
12 "worker_id": "k5:8085"
12 "worker_id": "k6:8086"
11 "worker_id": "k7:8087"
12 "worker_id": "k8:8088"
12 "worker_id": "k9:8089"
{code}
Regards
was (Author: yazgoo):
Hello, I launch the attached script :
[^rebalance.sh]
And in one of my test I get onne connect well balanced
{code:java}
❯ curl -s http://localhost:8081/connectors/s3-connector2/status | jq .tasks
|grep worker_id | sort | uniq -c
15 "worker_id": "k2:8082"
15 "worker_id": "k3:8083"
15 "worker_id": "k4:8084"
15 "worker_id": "k5:8085"
15 "worker_id": "k6:8086"
15 "worker_id": "k7:8087"
15 "worker_id": "k8:8088"
15 "worker_id": "k9:8089"
{code}
And the other one unbalanced
{code:java}
❯ curl -s http://localhost:8081/connectors/s3-connector1/status | jq .tasks
|grep worker_id | sort | uniq -c
27 "worker_id": "k1:8081"
11 "worker_id": "k2:8082"
12 "worker_id": "k3:8083"
11 "worker_id": "k4:8084"
12 "worker_id": "k5:8085"
12 "worker_id": "k6:8086"
11 "worker_id": "k7:8087"
12 "worker_id": "k8:8088"
12 "worker_id": "k9:8089"
{code}
Regards
> rebalancing leads to unevenly balanced connectors
> -------------------------------------------------
>
> Key: KAFKA-10413
> URL: https://issues.apache.org/jira/browse/KAFKA-10413
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Affects Versions: 2.5.1
> Reporter: yazgoo
> Assignee: rameshkrishnan muthusamy
> Priority: Major
> Fix For: 2.4.2, 2.5.2, 2.8.0, 2.7.1, 2.6.2
>
> Attachments: connect_worker_balanced.png, rebalance.sh
>
>
> GHi,
> With CP 5.5, running kafka connect s3 sink on EC2 whith autoscaling enabled,
> if a connect instance disappear, or a new one appear, we're seeing unbalanced
> consumption, much like mentionned in this post:
> [https://stackoverflow.com/questions/58644622/incremental-cooperative-rebalancing-leads-to-unevenly-balanced-connectors]
> This usually leads to one kafka connect instance taking most of the load and
> consumption not being able to keep on.
> Currently, we're "fixing" this by deleting the connector and re-creating it,
> but this is far from ideal.
> Any suggestion on what we could do to mitigate this ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
