Hi, I am wondering if QoS on handler threads is being looked at by someone ? We have found huge gains by doing this for asymmetric workloads which are extremely write heavy but latency on reads matters the most. Basically we forked out a separate thread pool of X threads for handling reads to prevent starvation of read requests which were far outnumbered by the # of writes.
As of now, we have been making adhoc changes (like introduce higher priority levels for certain types of requests like gets and scans) but having a framework to control this would be a really nice thing. One simple way to achieve this would be to have a group of "high priority handler threads" and be able to mark each table to prioritize "reads/writes". On receiving an RPC, the request is directly dispatched to this other pool of threads and the remaining RPCs go to the regular pool. This would be a naive implementation. The issue with this could be that what happens when the user ends up directing most of their load to the high priority pool which has fewer threads. We could do something simple like having a tight upper bound on the call queue length and if a new high priority call is rejected from this pool, just enqueue it to the regular pool of requests. Thoughts ? Thanks ! Varun
