[
https://issues.apache.org/jira/browse/KAFKA-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506085#comment-14506085
]
Jay Kreps commented on KAFKA-2139:
----------------------------------
Do you want to sketch out the design you have in mind? I really want to make
sure we don't add weird "business logic" inside the network layer if at all
possible. I also think that for our sanity it is important to maintain the
property that requests from a single connection are always processed in order
(which adding to the end of the queue would violate). I agree that we want it
to be the case that all traffic from the controller is prioritized over user
requests but let's find a generic way to do this as a feature of the network
layer without blurring those layers if possible.
> Add a separate controller messge queue with higher priority on broker side
> ---------------------------------------------------------------------------
>
> Key: KAFKA-2139
> URL: https://issues.apache.org/jira/browse/KAFKA-2139
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
>
> This ticket is supposed to be working together with KAFKA-2029.
> There are two issues with current controller to broker messages.
> 1. On the controller side the message are sent without synchronization.
> 2. On broker side the controller messages share the same queue as client
> messages.
> The problem here is that brokers process the controller messages for the same
> partition at different times and the variation could be big. This causes
> unnecessary data loss and prolong the preferred leader election / controlled
> shutdown/ partition reassignment, etc.
> KAFKA-2029 was trying to add a boundary between messages for different
> partitions. For example, before leader migration for previous partition
> finishes, the leader migration for next partition won't begin.
> This ticket is trying to let broker process controller messages faster. So
> the idea is have separate queue to hold controller messages, if there are
> controller messages, KafkaApi thread will first take care of those messages,
> otherwise it will proceed messages from clients.
> Those two tickets are not ultimate solution to current controller problems,
> but just mitigate them with minor code changes. Moving forward, we still need
> to think about rewriting controller in a cleaner way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
