This thread dump has been taken when client was sending 60 requests in parallel (at least, in theory). There are 50 server handler threads.
On Tue, Jul 30, 2013 at 1:15 PM, Vladimir Rodionov <[email protected]>wrote: > Sure, here it is: > > http://pastebin.com/8TjyrKRT > > epoll is not only to read/write HDFS but to connect/listen to clients as > well? > > > On Tue, Jul 30, 2013 at 12:31 PM, Jean-Daniel Cryans > <[email protected]>wrote: > >> Can you show us what the thread dump looks like when the threads are >> BLOCKED? There aren't that many locks on the read path when reading >> out of the block cache, and epoll would only happen if you need to hit >> HDFS, which you're saying is not happening. >> >> J-D >> >> On Tue, Jul 30, 2013 at 12:16 PM, Vladimir Rodionov >> <[email protected]> wrote: >> > I am hitting data in a block cache, of course. The data set is very >> small >> > to fit comfortably into block cache and all request are directed to the >> > same Region to guarantee single RS testing. >> > >> > To Ted: >> > >> > Yes, its CDH 4.3 . What the difference between 94.10 and 94.6 with >> respect >> > to read performance? >> > >> > >> > On Tue, Jul 30, 2013 at 12:06 PM, Jean-Daniel Cryans < >> [email protected]>wrote: >> > >> >> That's a tough one. >> >> >> >> One thing that comes to mind is socket reuse. It used to come up more >> >> more often but this is an issue that people hit when doing loads of >> >> random reads. Try enabling tcp_tw_recycle but I'm not guaranteeing >> >> anything :) >> >> >> >> Also if you _just_ want to saturate something, be it CPU or network, >> >> wouldn't it be better to hit data only in the block cache? This way it >> >> has the lowest overhead? >> >> >> >> Last thing I wanted to mention is that yes, the client doesn't scale >> >> very well. I would suggest you give the asynchbase client a run. >> >> >> >> J-D >> >> >> >> On Tue, Jul 30, 2013 at 11:23 AM, Vladimir Rodionov >> >> <[email protected]> wrote: >> >> > I have been doing quite extensive testing of different read >> scenarios: >> >> > >> >> > 1. blockcache disabled/enabled >> >> > 2. data is local/remote (no good hdfs locality) >> >> > >> >> > and it turned out that that I can not saturate 1 RS using one >> >> (comparable in CPU power and RAM) client host: >> >> > >> >> > I am running client app with 60 read threads active (with multi-get) >> >> that is going to one particular RS and >> >> > this RS's load is 100 -150% (out of 3200% available) - it means that >> >> load is ~5% >> >> > >> >> > All threads in RS are either in BLOCKED (wait) or in IN_NATIVE states >> >> (epoll) >> >> > >> >> > I attribute this to the HBase client implementation which seems to >> be >> >> not scalable (I am going dig into client later on today). >> >> > >> >> > Some numbers: The maximum what I could get from Single get (60 >> threads): >> >> 30K per sec. Multiget gives ~ 75K (60 threads) >> >> > >> >> > What are my options? I want to measure the limits and I do not want >> to >> >> run Cluster of clients against just ONE Region Server? >> >> > >> >> > RS config: 96GB RAM, 16(32) CPU >> >> > Client : 48GB RAM 8 (16) CPU >> >> > >> >> > Best regards, >> >> > Vladimir Rodionov >> >> > Principal Platform Engineer >> >> > Carrier IQ, www.carrieriq.com >> >> > e-mail: [email protected] >> >> > >> >> > >> >> > Confidentiality Notice: The information contained in this message, >> >> including any attachments hereto, may be confidential and is intended >> to be >> >> read only by the individual or entity to whom this message is >> addressed. If >> >> the reader of this message is not the intended recipient or an agent or >> >> designee of the intended recipient, please note that any review, use, >> >> disclosure or distribution of this message or its attachments, in any >> form, >> >> is strictly prohibited. If you have received this message in error, >> please >> >> immediately notify the sender and/or [email protected] and >> >> delete or destroy any copy of this message and its attachments. >> >> >> > >
