In hdfs-site.xml, can you check the values for the following config ? <property> <name>ipc.server.tcpnodelay</name> <value>true</value> </property> <property> <name>ipc.client.tcpnodelay</name> <value>true</value> </property>
On Tue, Jul 30, 2013 at 1:58 PM, Vladimir Rodionov <[email protected]>wrote: > With : > > hbase.ipc.client.tcpnodelay= true > hbase.client.ipc.pool.size =5 > > I was able to achieve 50K per sec for single get operations. No progress > for multi-gets. > > > On Tue, Jul 30, 2013 at 1:52 PM, Vladimir Rodionov > <[email protected]>wrote: > > > Exactly, but this thread dump is from RS under load nevertheless (you can > > see that one thread is in JAVA and reading data from socket) > > > > > > On Tue, Jul 30, 2013 at 1:35 PM, Jean-Daniel Cryans <[email protected] > >wrote: > > > >> FWIW nothing is happening in that thread dump. > >> > >> J-D > >> > >> On Tue, Jul 30, 2013 at 1:15 PM, Vladimir Rodionov > >> <[email protected]> wrote: > >> > Sure, here it is: > >> > > >> > http://pastebin.com/8TjyrKRT > >> > > >> > epoll is not only to read/write HDFS but to connect/listen to clients > as > >> > well? > >> > > >> > > >> > On Tue, Jul 30, 2013 at 12:31 PM, Jean-Daniel Cryans < > >> [email protected]>wrote: > >> > > >> >> Can you show us what the thread dump looks like when the threads are > >> >> BLOCKED? There aren't that many locks on the read path when reading > >> >> out of the block cache, and epoll would only happen if you need to > hit > >> >> HDFS, which you're saying is not happening. > >> >> > >> >> J-D > >> >> > >> >> On Tue, Jul 30, 2013 at 12:16 PM, Vladimir Rodionov > >> >> <[email protected]> wrote: > >> >> > I am hitting data in a block cache, of course. The data set is very > >> small > >> >> > to fit comfortably into block cache and all request are directed to > >> the > >> >> > same Region to guarantee single RS testing. > >> >> > > >> >> > To Ted: > >> >> > > >> >> > Yes, its CDH 4.3 . What the difference between 94.10 and 94.6 with > >> >> respect > >> >> > to read performance? > >> >> > > >> >> > > >> >> > On Tue, Jul 30, 2013 at 12:06 PM, Jean-Daniel Cryans < > >> >> [email protected]>wrote: > >> >> > > >> >> >> That's a tough one. > >> >> >> > >> >> >> One thing that comes to mind is socket reuse. It used to come up > >> more > >> >> >> more often but this is an issue that people hit when doing loads > of > >> >> >> random reads. Try enabling tcp_tw_recycle but I'm not guaranteeing > >> >> >> anything :) > >> >> >> > >> >> >> Also if you _just_ want to saturate something, be it CPU or > network, > >> >> >> wouldn't it be better to hit data only in the block cache? This > way > >> it > >> >> >> has the lowest overhead? > >> >> >> > >> >> >> Last thing I wanted to mention is that yes, the client doesn't > scale > >> >> >> very well. I would suggest you give the asynchbase client a run. > >> >> >> > >> >> >> J-D > >> >> >> > >> >> >> On Tue, Jul 30, 2013 at 11:23 AM, Vladimir Rodionov > >> >> >> <[email protected]> wrote: > >> >> >> > I have been doing quite extensive testing of different read > >> scenarios: > >> >> >> > > >> >> >> > 1. blockcache disabled/enabled > >> >> >> > 2. data is local/remote (no good hdfs locality) > >> >> >> > > >> >> >> > and it turned out that that I can not saturate 1 RS using one > >> >> >> (comparable in CPU power and RAM) client host: > >> >> >> > > >> >> >> > I am running client app with 60 read threads active (with > >> multi-get) > >> >> >> that is going to one particular RS and > >> >> >> > this RS's load is 100 -150% (out of 3200% available) - it means > >> that > >> >> >> load is ~5% > >> >> >> > > >> >> >> > All threads in RS are either in BLOCKED (wait) or in IN_NATIVE > >> states > >> >> >> (epoll) > >> >> >> > > >> >> >> > I attribute this to the HBase client implementation which seems > >> to be > >> >> >> not scalable (I am going dig into client later on today). > >> >> >> > > >> >> >> > Some numbers: The maximum what I could get from Single get (60 > >> >> threads): > >> >> >> 30K per sec. Multiget gives ~ 75K (60 threads) > >> >> >> > > >> >> >> > What are my options? I want to measure the limits and I do not > >> want to > >> >> >> run Cluster of clients against just ONE Region Server? > >> >> >> > > >> >> >> > RS config: 96GB RAM, 16(32) CPU > >> >> >> > Client : 48GB RAM 8 (16) CPU > >> >> >> > > >> >> >> > Best regards, > >> >> >> > Vladimir Rodionov > >> >> >> > Principal Platform Engineer > >> >> >> > Carrier IQ, www.carrieriq.com > >> >> >> > e-mail: [email protected] > >> >> >> > > >> >> >> > > >> >> >> > Confidentiality Notice: The information contained in this > >> message, > >> >> >> including any attachments hereto, may be confidential and is > >> intended > >> >> to be > >> >> >> read only by the individual or entity to whom this message is > >> >> addressed. If > >> >> >> the reader of this message is not the intended recipient or an > >> agent or > >> >> >> designee of the intended recipient, please note that any review, > >> use, > >> >> >> disclosure or distribution of this message or its attachments, in > >> any > >> >> form, > >> >> >> is strictly prohibited. If you have received this message in > error, > >> >> please > >> >> >> immediately notify the sender and/or > [email protected] > >> >> >> delete or destroy any copy of this message and its attachments. > >> >> >> > >> >> > >> > > > > >
