[
https://issues.apache.org/jira/browse/KAFKA-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513534#comment-14513534
]
Jiangjie Qin commented on KAFKA-2139:
-------------------------------------
Hey Gwen, thanks for this really quick review!
Yes, I mean NetworkClient instead of SocketServer.
Very good point about a separate controller port! I actually have always been a
big fan of separating the control plain with data plain. Unfortunately simply
having a separate port for controller probably won't help. Currently the
connections are always initiated from controller to broker, not the other way.
This makes sense because it is the controller who wants to send requests to
brokers.
To really separate controller traffic, we have to separate controller message
processing from KafkaApi completely on broker. So all the messages from user
port will be processed by KafkaClientApi and controller messages will be taken
care of by KafkaControllerApi. KafkaControllerApi will only take messages from
the controller-to-broker connection but not other clients' connections.
I think what [~toddpalino] actually meant is to separate replication traffic
from producing/consuming traffic. That seems orthogonal to this change.
> Add a separate controller messge queue with higher priority on broker side
> ---------------------------------------------------------------------------
>
> Key: KAFKA-2139
> URL: https://issues.apache.org/jira/browse/KAFKA-2139
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
>
> This ticket is supposed to be working together with KAFKA-2029.
> There are two issues with current controller to broker messages.
> 1. On the controller side the message are sent without synchronization.
> 2. On broker side the controller messages share the same queue as client
> messages.
> The problem here is that brokers process the controller messages for the same
> partition at different times and the variation could be big. This causes
> unnecessary data loss and prolong the preferred leader election / controlled
> shutdown/ partition reassignment, etc.
> KAFKA-2029 was trying to add a boundary between messages for different
> partitions. For example, before leader migration for previous partition
> finishes, the leader migration for next partition won't begin.
> This ticket is trying to let broker process controller messages faster. So
> the idea is have separate queue to hold controller messages, if there are
> controller messages, KafkaApi thread will first take care of those messages,
> otherwise it will proceed messages from clients.
> Those two tickets are not ultimate solution to current controller problems,
> but just mitigate them with minor code changes. Moving forward, we still need
> to think about rewriting controller in a cleaner way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
