Network? 1GbE or 10GbE? Sent from a remote device. Please excuse any typos...
Mike Segel On Jul 31, 2013, at 9:27 PM, "Vladimir Rodionov" <[email protected]> wrote: > Some final numbers : > > Test config: > > HBase 0.94.6 > blockcache=true, block size = 64K, KV size = 62 bytes (raw). > > 5 Clients: 96GB, 16(32) CPUs (2.2Ghz), CentOS 5.7 > 1 RS Server: the same config. > > Local network with ping between hosts: 0.1 ms > > > 1. HBase client hits the wall at ~ 50K per sec regardless of # of CPU, > threads, IO pool size and other settings. > 2. HBase server was able to sustain 170K per sec (with 64K block size). All > from block cache. KV size = 62 bytes (very small). This is for single Get > op, 60 threads per client, 5 clients (on different hosts) > 3. Multi - get hits the wall at the same 170K-200K per sec. Batch size > tested: 30, 100. The same performance absolutely as with batch size = 1. > Multi get has some internal issues on RegionServer side. May be excessive > locking or some thing else. > > > > > > On Tue, Jul 30, 2013 at 2:01 PM, Vladimir Rodionov > <[email protected]>wrote: > >> 1. SCR are enabled >> 2. Single Configuration for all table did not work well, but I will try it >> again >> 3. With Nagel I had 0.8ms avg, w/o - 0.4ms - I see the difference >> >> >> On Tue, Jul 30, 2013 at 1:50 PM, lars hofhansl <[email protected]> wrote: >> >>> With Nagle's you'd see something around 40ms. You are not saying 0.8ms >>> RTT is bad, right? Are you seeing ~40ms latencies? >>> >>> This thread has gotten confusing. >>> >>> I would try these: >>> * one Configuration for all tables. Or even use a single >>> HConnection/Threadpool and use the HTable(byte[], HConnection, >>> ExecutorService) constructor >>> * disable Nagle's: set both ipc.server.tcpnodelay and >>> hbase.ipc.client.tcpnodelay to true in hbase-site.xml (both client *and* >>> server) >>> * increase hbase.client.ipc.pool.size in client's hbase-site.xml >>> * enable short circuit reads (details depend on exact version of Hadoop). >>> Google will help :) >>> >>> -- Lars >>> >>> >>> ----- Original Message ----- >>> From: Vladimir Rodionov <[email protected]> >>> To: [email protected] >>> Cc: >>> Sent: Tuesday, July 30, 2013 1:30 PM >>> Subject: Re: HBase read perfomnance and HBase client >>> >>> This hbase.ipc.client.tcpnodelay (default - false) explains poor single >>> thread performance and high latency ( 0.8ms in local network)? >>> >>> >>> On Tue, Jul 30, 2013 at 1:22 PM, Vladimir Rodionov >>> <[email protected]>wrote: >>> >>>> One more observation: One Configuration instance per HTable gives 50% >>>> boost as compared to single Configuration object for all HTable's - from >>>> 20K to 30K >>>> >>>> >>>> On Tue, Jul 30, 2013 at 1:17 PM, Vladimir Rodionov < >>> [email protected] >>>>> wrote: >>>> >>>>> This thread dump has been taken when client was sending 60 requests in >>>>> parallel (at least, in theory). There are 50 server handler threads. >>>>> >>>>> >>>>> On Tue, Jul 30, 2013 at 1:15 PM, Vladimir Rodionov < >>>>> [email protected]> wrote: >>>>> >>>>>> Sure, here it is: >>>>>> >>>>>> http://pastebin.com/8TjyrKRT >>>>>> >>>>>> epoll is not only to read/write HDFS but to connect/listen to clients >>> as >>>>>> well? >>>>>> >>>>>> >>>>>> On Tue, Jul 30, 2013 at 12:31 PM, Jean-Daniel Cryans < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Can you show us what the thread dump looks like when the threads are >>>>>>> BLOCKED? There aren't that many locks on the read path when reading >>>>>>> out of the block cache, and epoll would only happen if you need to >>> hit >>>>>>> HDFS, which you're saying is not happening. >>>>>>> >>>>>>> J-D >>>>>>> >>>>>>> On Tue, Jul 30, 2013 at 12:16 PM, Vladimir Rodionov >>>>>>> <[email protected]> wrote: >>>>>>>> I am hitting data in a block cache, of course. The data set is very >>>>>>> small >>>>>>>> to fit comfortably into block cache and all request are directed to >>>>>>> the >>>>>>>> same Region to guarantee single RS testing. >>>>>>>> >>>>>>>> To Ted: >>>>>>>> >>>>>>>> Yes, its CDH 4.3 . What the difference between 94.10 and 94.6 with >>>>>>> respect >>>>>>>> to read performance? >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 30, 2013 at 12:06 PM, Jean-Daniel Cryans < >>>>>>> [email protected]>wrote: >>>>>>>> >>>>>>>>> That's a tough one. >>>>>>>>> >>>>>>>>> One thing that comes to mind is socket reuse. It used to come up >>> more >>>>>>>>> more often but this is an issue that people hit when doing loads >>> of >>>>>>>>> random reads. Try enabling tcp_tw_recycle but I'm not guaranteeing >>>>>>>>> anything :) >>>>>>>>> >>>>>>>>> Also if you _just_ want to saturate something, be it CPU or >>> network, >>>>>>>>> wouldn't it be better to hit data only in the block cache? This >>> way >>>>>>> it >>>>>>>>> has the lowest overhead? >>>>>>>>> >>>>>>>>> Last thing I wanted to mention is that yes, the client doesn't >>> scale >>>>>>>>> very well. I would suggest you give the asynchbase client a run. >>>>>>>>> >>>>>>>>> J-D >>>>>>>>> >>>>>>>>> On Tue, Jul 30, 2013 at 11:23 AM, Vladimir Rodionov >>>>>>>>> <[email protected]> wrote: >>>>>>>>>> I have been doing quite extensive testing of different read >>>>>>> scenarios: >>>>>>>>>> >>>>>>>>>> 1. blockcache disabled/enabled >>>>>>>>>> 2. data is local/remote (no good hdfs locality) >>>>>>>>>> >>>>>>>>>> and it turned out that that I can not saturate 1 RS using one >>>>>>>>> (comparable in CPU power and RAM) client host: >>>>>>>>>> >>>>>>>>>> I am running client app with 60 read threads active (with >>>>>>> multi-get) >>>>>>>>> that is going to one particular RS and >>>>>>>>>> this RS's load is 100 -150% (out of 3200% available) - it means >>>>>>> that >>>>>>>>> load is ~5% >>>>>>>>>> >>>>>>>>>> All threads in RS are either in BLOCKED (wait) or in IN_NATIVE >>>>>>> states >>>>>>>>> (epoll) >>>>>>>>>> >>>>>>>>>> I attribute this to the HBase client implementation which seems >>>>>>> to be >>>>>>>>> not scalable (I am going dig into client later on today). >>>>>>>>>> >>>>>>>>>> Some numbers: The maximum what I could get from Single get (60 >>>>>>> threads): >>>>>>>>> 30K per sec. Multiget gives ~ 75K (60 threads) >>>>>>>>>> >>>>>>>>>> What are my options? I want to measure the limits and I do not >>>>>>> want to >>>>>>>>> run Cluster of clients against just ONE Region Server? >>>>>>>>>> >>>>>>>>>> RS config: 96GB RAM, 16(32) CPU >>>>>>>>>> Client : 48GB RAM 8 (16) CPU >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Vladimir Rodionov >>>>>>>>>> Principal Platform Engineer >>>>>>>>>> Carrier IQ, www.carrieriq.com >>>>>>>>>> e-mail: [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Confidentiality Notice: The information contained in this >>> message, >>>>>>>>> including any attachments hereto, may be confidential and is >>>>>>> intended to be >>>>>>>>> read only by the individual or entity to whom this message is >>>>>>> addressed. If >>>>>>>>> the reader of this message is not the intended recipient or an >>> agent >>>>>>> or >>>>>>>>> designee of the intended recipient, please note that any review, >>> use, >>>>>>>>> disclosure or distribution of this message or its attachments, in >>>>>>> any form, >>>>>>>>> is strictly prohibited. If you have received this message in >>> error, >>>>>>> please >>>>>>>>> immediately notify the sender and/or [email protected] >>>>>>>>> delete or destroy any copy of this message and its attachments. >>
