This does make sense, but I wonder if it would be very usable or worth the effort? Maybe the solution is usually to just have good monitoring and have sufficient capacity for your workload--that is rather than trying to handle overload more gracefully, just avoid it by adding servers.
In any case, here would be one implementation approach: Currently (as of 0.8) we keep a queue of requests waiting for processing by one of N I/O threads (RequestChannel.scala). Currently this is just a linear blocking queue. We could make it into a priority queue and assign a priority to each request (either based on the topic or on a per-request basis). Instead of processing requests on a FIFO basis we would always process the highest priority request. The problem with this, of course, is that it introduces all the scheduling problems common to OS resource schedulers. For example if there is a continuous stream of high priority requests it will starve out the lower priority request from EVER getting processed, which may or may not be desired. This also complicates the ordering guarantees. Since now two messages for the same topic/partition would only be appended in-order if they are of exactly the same priority. -Jay On Mon, Jun 11, 2012 at 5:08 PM, xiaoyu wang <xiaoyu.w...@gmail.com> wrote: > Hi all, > > Just want to bring up some discussion, does it make sense to have priority > defined over topics? In case of resource contention, topic with higher > priority get delivered first. > > > -Xiaoyu >
