[
https://issues.apache.org/jira/browse/KAFKA-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-7121:
--------------------------------
Description:
We started a connector, and even though it is in RUNNING status, tasks are not
getting assigned:
{{
{"name":"prod-xxx-v2","connector":}}{{{"state":"RUNNING","worker_id":"[0.0.0.0:8083|http://0.0.0.0:8083/]"}}}{{,"tasks":[],"type":"sink"}}}
Other connectors are running without issues.
Attempt to restart the connector returned 409 status.
Logs show the following messages, keep repeating for hours:
{{{{ [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2
failed unexpectedly, this connector will not be properly reconfigured unless
manually triggered.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:956)}}}}
{{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
{{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
{{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
{{ {{ [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure
connector tasks failed
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018)}}}}
{{ \{{ org.apache.kafka.connect.runtime.rest.errors.ConnectRestException:
Cannot complete request because of a conflicting operation (e.g. worker
rebalance)}}}}
{{ \{{ at
org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)}}}}
{{ \{{ at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015)}}}}
{{ \{{ at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}}}
{{ \{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}}}
{{ \{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}}}
{{ \{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}}}
{{ \{{ at java.lang.Thread.run(Thread.java:748)}}}}
was:
We started a connector, and even though it is in RUNNING status, tasks are not
getting assigned:
{"name":"prod-xxx-v2","connector":
{"state":"RUNNING","worker_id":"[0.0.0.0:8083|http://0.0.0.0:8083/]"}
,"tasks":[],"type":"sink"}
Other connectors are running without issues.
Attempt to restart the connector returned 409 status.
Logs show the following messages, keep repeating for hours:
{{ [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2 failed
unexpectedly, this connector will not be properly reconfigured unless manually
triggered.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:956)}}
{{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}
{{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}
{{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
+0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)}}
{{ [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure connector
tasks failed
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018)}}
{{ org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot
complete request because of a conflicting operation (e.g. worker rebalance)}}
{{ at
org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)}}
{{ at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
> Intermittently, Connectors fail to assign tasks and keep retrying every
> second forever.
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-7121
> URL: https://issues.apache.org/jira/browse/KAFKA-7121
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Gwen Shapira
> Priority: Major
>
> We started a connector, and even though it is in RUNNING status, tasks are
> not getting assigned:
> {{
> {"name":"prod-xxx-v2","connector":}}{{{"state":"RUNNING","worker_id":"[0.0.0.0:8083|http://0.0.0.0:8083/]"}}}{{,"tasks":[],"type":"sink"}}}
> Other connectors are running without issues.
> Attempt to restart the connector returned 409 status.
> Logs show the following messages, keep repeating for hours:
> {{{{ [2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2
> failed unexpectedly, this connector will not be properly reconfigured unless
> manually triggered.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:956)}}}}
> {{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
> +0000] "POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0
> (org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
> {{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
> +0000] "POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1
> (org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
> {{ {{ [2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19
> +0000] "POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1
> (org.apache.kafka.connect.runtime.rest.RestServer:60)}}}}
> {{ {{ [2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure
> connector tasks failed
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018)}}}}
> {{ \{{ org.apache.kafka.connect.runtime.rest.errors.ConnectRestException:
> Cannot complete request because of a conflicting operation (e.g. worker
> rebalance)}}}}
> {{ \{{ at
> org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)}}}}
> {{ \{{ at
> org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015)}}}}
> {{ \{{ at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}}}
> {{ \{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}}}
> {{ \{{ at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}}}
> {{ \{{ at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}}}
> {{ \{{ at java.lang.Thread.run(Thread.java:748)}}}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
