I am runing expriment at the following scenario : both client-1 and client-2 send update requests, the client-1 write the large objects(100KB record) into table-1, and client-2 write the small objects (1KB record) into table-2. The charts below shows the effect of the Mutilple-Type queue feature:
[image: 内嵌图片 1] as shown in future 1(the default version of hbase 1.1.2): 1. in the beginning, client-2 starts to sends requests and client-1 not. the throughput of client-2 keep stable 2. when client-1 starts, the throughputs of both client-1 and client-2 are very unstable, that is to say client-1 influence client-2 seriously 3. the average hbase throughput is 36641 . [image: 内嵌图片 4] Afterwords, we assign 150 handlers to table-2 and 3 handlers to table-1 with mutilple-type queue feature. as shown in future 2, the throughputs of both client-1 and client-2 are more smooth, the average hbase throughput is 51653 , 41% higher. so we think in some cases the feature is a improvement for running multiple workloads on a single hbase cluster, let me know if anything is incorrect or inaccurate . Thanks a lot for your help! 2017-04-11 13:58 GMT+08:00 范范欣欣 <[email protected]>: > Now, the feature HBASE-11355 seperates the single Call Queue into > MutilQueue(get call queue, write call queue and scan call queue), and each > type queue can specify fixed number of handlers. It's helpful in some > outages , to avoid all read or all write requests ran out of handler > threads. > > however, there are still several problems : > > 1. workloads in the same request type(get/write/scan) may influence each > other as before, consider the following scenario: > > (1) both client-1 and client-2 send write requests, the client-1 write the > large objects(100KB record) , and client-2 write the small objects (1KB > record). the client -1 will ran out of all handler threads of the > write-queue, and decrease the client-2 throughput > > (2) both client-3 and client-4 send get requests, the client-3 search all > data from lots of hfiles( all search key are equally popular), read latency > is high. the client-4 do not require any I/O resources(say, data is > cached). the client-3 will ran out of all handler threads of the get-queue, > and increase the read latency of client-4 > > 2. administor can't increate/decrease the handler number for the specified > queue easily > > > we are trying to implement a new Mutilple-Typed Queue, administor can > create a queue with a specified number of handler for specified table and > specified request type(get/write/scan), as: > > create_queue 'queue1' ,{'handler' => 100} > > grant_queue 'table1','scan','queue1' > > grant_queue 'tableN','scan','queue1' > > > create_queue 'queue2' ,{'handler' => 50} > > grant_queue 'table2','write','queue2' > > > the idea based on the fact that the workload for a specified table and > request type will be unique. > > in addition, administor can manager the queue with commands: > > //easily increase/decrease handlers > > alter_queue 'queue1' ,{'handler' => 50} > > //list all queues > > list_queues > > //drop the specified queue > > drop_queue 'queue1' > > > I am wondering if the developers could look at the idea and let me know if > anything is incorrect or inaccurate, or if I have missed anything. > > > Thanks a lot for your help! >
