[
https://issues.apache.org/jira/browse/KAFKA-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jozef Vilcek updated KAFKA-13196:
---------------------------------
Description:
I am using MirrorMaker 2.0 and running it via [
MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
class. This method will start up `DistributedHerder` and will use
non-functional `advertisedUrl`, and therefore workers can not talk to each
other and coordinate.
After upgrading from version `2.4.0` to `2.7.1` I noticed that when I am
starting up mirror maker it does not always start tasks - just connector is
running. Doing some amount of stop/starts will eventually start tasks too.
After a bit of digging I did notice that in attempt to configure connector's
task, code ends up in this [1] branch, where configure request is being
forwarded to the leader. For some reason, task configuration is not done on
leader. However, MirrorMaker does not pack RestServer and therefore that
request will never succeed.
I am not sure what is going no or why it does seem to work better on 2.4.0. I
noticed that connector start procedure did involve less callbacks on 2.4.0 in
connector start sequence [2].
[1]
[https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494]
[2]
[https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236]
was:
I am using MirrorMaker 2.0 and running it via [
MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
class. This method will start up `DistributedHerder` and will use
non-functional `advertisedUrl`, and therefore workers can not talk to each
other and coordinate.
After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am
starting up mirror maker ti does not always start tasks - just connector is
executing. Doing some amount of stop and start it will eventually start tasks
too.
After a bit of digging I did noticed that in attempt to configure connector's
task, code ends up in this [1] branch, where configure request is being
forwarded to the leader. From some reason, task configuration is not done on
leader. However, MirrorMaker does not pack RestServer and therefore that
request will never succeed.
I am not sure what is going no or why it does seem to work better on 2.4.0. I
noticed that connector start procedure did involve less callbacks on 2.4.0 in
connector start sequence [2].
[1]
[https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494]
[2]
[https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236]
> MirrorMaker 2 not always start tasks
> ------------------------------------
>
> Key: KAFKA-13196
> URL: https://issues.apache.org/jira/browse/KAFKA-13196
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.7.1
> Reporter: Jozef Vilcek
> Priority: Major
>
> I am using MirrorMaker 2.0 and running it via [
> MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
> class. This method will start up `DistributedHerder` and will use
> non-functional `advertisedUrl`, and therefore workers can not talk to each
> other and coordinate.
> After upgrading from version `2.4.0` to `2.7.1` I noticed that when I am
> starting up mirror maker it does not always start tasks - just connector is
> running. Doing some amount of stop/starts will eventually start tasks too.
> After a bit of digging I did notice that in attempt to configure connector's
> task, code ends up in this [1] branch, where configure request is being
> forwarded to the leader. For some reason, task configuration is not done on
> leader. However, MirrorMaker does not pack RestServer and therefore that
> request will never succeed.
> I am not sure what is going no or why it does seem to work better on 2.4.0. I
> noticed that connector start procedure did involve less callbacks on 2.4.0 in
> connector start sequence [2].
>
>
> [1]
> [https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494]
> [2]
> [https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
