All tests I have run were hitting single region on a region server. I
suspect this is not a right scenario. There are some points in the Store
class which are heavily synchronized:
For example this one:
// All access must be synchronized.
private final CopyOnWriteArraySet<ChangedReadersObserver>
changedReaderObservers =
new CopyOnWriteArraySet<ChangedReadersObserver>();
I will re-run tests against all available regions on a RS and will post
results later on today.
On Wed, Jul 31, 2013 at 11:15 PM, lars hofhansl <[email protected]> wrote:
> Yeah, that would seem to indicate that seeking into the block is not a
> bottleneck (and you said earlier that everything fits into the blockcache).
> Need to profile to know more. If you have time, would be cool if you can
> start jvisualvm and attach it to the RS start the profiling and let the
> workload run for a bit.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Vladimir Rodionov <[email protected]>
> To: [email protected]; lars hofhansl <[email protected]>
> Cc:
> Sent: Wednesday, July 31, 2013 9:57 PM
> Subject: Re: HBase read perfomnance and HBase client
>
> Smaller block size (32K) does not give any performance gain and this is
> strange, to say the least.
>
>
> On Wed, Jul 31, 2013 at 9:33 PM, lars hofhansl <[email protected]> wrote:
>
> > Would be interesting to profile MultiGet. With RTT of 0.1ms, the internal
> > RS friction is probably the main contributor.
> > In fact MultiGet just loops over the set at the RS and calls single gets
> > on the various regions.
> >
> > Each Get needs to reseek into the block (even when it is cached, since
> KVs
> > have variable size).
> >
> > There are HBASE-6136 and HBASE-8362.
> >
> >
> > -- Lars
> >
> > ________________________________
> > From: Vladimir Rodionov <[email protected]>
> > To: [email protected]; lars hofhansl <[email protected]>
> > Sent: Wednesday, July 31, 2013 7:27 PM
> > Subject: Re: HBase read perfomnance and HBase client
> >
> >
> > Some final numbers :
> >
> > Test config:
> >
> > HBase 0.94.6
> > blockcache=true, block size = 64K, KV size = 62 bytes (raw).
> >
> > 5 Clients: 96GB, 16(32) CPUs (2.2Ghz), CentOS 5.7
> > 1 RS Server: the same config.
> >
> > Local network with ping between hosts: 0.1 ms
> >
> >
> > 1. HBase client hits the wall at ~ 50K per sec regardless of # of CPU,
> > threads, IO pool size and other settings.
> > 2. HBase server was able to sustain 170K per sec (with 64K block size).
> All
> > from block cache. KV size = 62 bytes (very small). This is for single Get
> > op, 60 threads per client, 5 clients (on different hosts)
> > 3. Multi - get hits the wall at the same 170K-200K per sec. Batch size
> > tested: 30, 100. The same performance absolutely as with batch size = 1.
> > Multi get has some internal issues on RegionServer side. May be excessive
> > locking or some thing else.
> >
> >
> >
> >
> >
> > On Tue, Jul 30, 2013 at 2:01 PM, Vladimir Rodionov
> > <[email protected]>wrote:
> >
> > > 1. SCR are enabled
> > > 2. Single Configuration for all table did not work well, but I will try
> > it
> > > again
> > > 3. With Nagel I had 0.8ms avg, w/o - 0.4ms - I see the difference
> > >
> > >
> > > On Tue, Jul 30, 2013 at 1:50 PM, lars hofhansl <[email protected]>
> wrote:
> > >
> > >> With Nagle's you'd see something around 40ms. You are not saying 0.8ms
> > >> RTT is bad, right? Are you seeing ~40ms latencies?
> > >>
> > >> This thread has gotten confusing.
> > >>
> > >> I would try these:
> > >> * one Configuration for all tables. Or even use a single
> > >> HConnection/Threadpool and use the HTable(byte[], HConnection,
> > >> ExecutorService) constructor
> > >> * disable Nagle's: set both ipc.server.tcpnodelay and
> > >> hbase.ipc.client.tcpnodelay to true in hbase-site.xml (both client
> *and*
> > >> server)
> > >> * increase hbase.client.ipc.pool.size in client's hbase-site.xml
> > >> * enable short circuit reads (details depend on exact version of
> > Hadoop).
> > >> Google will help :)
> > >>
> > >> -- Lars
> > >>
> > >>
> > >> ----- Original Message -----
> > >> From: Vladimir Rodionov <[email protected]>
> > >> To: [email protected]
> > >> Cc:
> > >> Sent: Tuesday, July 30, 2013 1:30 PM
> > >> Subject: Re: HBase read perfomnance and HBase client
> > >>
> > >> This hbase.ipc.client.tcpnodelay (default - false) explains poor
> single
> > >> thread performance and high latency ( 0.8ms in local network)?
> > >>
> > >>
> > >> On Tue, Jul 30, 2013 at 1:22 PM, Vladimir Rodionov
> > >> <[email protected]>wrote:
> > >>
> > >> > One more observation: One Configuration instance per HTable gives
> 50%
> > >> > boost as compared to single Configuration object for all HTable's -
> > from
> > >> > 20K to 30K
> > >> >
> > >> >
> > >> > On Tue, Jul 30, 2013 at 1:17 PM, Vladimir Rodionov <
> > >> [email protected]
> > >> > > wrote:
> > >> >
> > >> >> This thread dump has been taken when client was sending 60 requests
> > in
> > >> >> parallel (at least, in theory). There are 50 server handler
> threads.
> > >> >>
> > >> >>
> > >> >> On Tue, Jul 30, 2013 at 1:15 PM, Vladimir Rodionov <
> > >> >> [email protected]> wrote:
> > >> >>
> > >> >>> Sure, here it is:
> > >> >>>
> > >> >>> http://pastebin.com/8TjyrKRT
> > >> >>>
> > >> >>> epoll is not only to read/write HDFS but to connect/listen to
> > clients
> > >> as
> > >> >>> well?
> > >> >>>
> > >> >>>
> > >> >>> On Tue, Jul 30, 2013 at 12:31 PM, Jean-Daniel Cryans <
> > >> >>> [email protected]> wrote:
> > >> >>>
> > >> >>>> Can you show us what the thread dump looks like when the threads
> > are
> > >> >>>> BLOCKED? There aren't that many locks on the read path when
> reading
> > >> >>>> out of the block cache, and epoll would only happen if you need
> to
> > >> hit
> > >> >>>> HDFS, which you're saying is not happening.
> > >> >>>>
> > >> >>>> J-D
> > >> >>>>
> > >> >>>> On Tue, Jul 30, 2013 at 12:16 PM, Vladimir Rodionov
> > >> >>>> <[email protected]> wrote:
> > >> >>>> > I am hitting data in a block cache, of course. The data set is
> > very
> > >> >>>> small
> > >> >>>> > to fit comfortably into block cache and all request are
> directed
> > to
> > >> >>>> the
> > >> >>>> > same Region to guarantee single RS testing.
> > >> >>>> >
> > >> >>>> > To Ted:
> > >> >>>> >
> > >> >>>> > Yes, its CDH 4.3 . What the difference between 94.10 and 94.6
> > with
> > >> >>>> respect
> > >> >>>> > to read performance?
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > On Tue, Jul 30, 2013 at 12:06 PM, Jean-Daniel Cryans <
> > >> >>>> [email protected]>wrote:
> > >> >>>> >
> > >> >>>> >> That's a tough one.
> > >> >>>> >>
> > >> >>>> >> One thing that comes to mind is socket reuse. It used to come
> up
> > >> more
> > >> >>>> >> more often but this is an issue that people hit when doing
> loads
> > >> of
> > >> >>>> >> random reads. Try enabling tcp_tw_recycle but I'm not
> > guaranteeing
> > >> >>>> >> anything :)
> > >> >>>> >>
> > >> >>>> >> Also if you _just_ want to saturate something, be it CPU or
> > >> network,
> > >> >>>> >> wouldn't it be better to hit data only in the block cache?
> This
> > >> way
> > >> >>>> it
> > >> >>>> >> has the lowest overhead?
> > >> >>>> >>
> > >> >>>> >> Last thing I wanted to mention is that yes, the client doesn't
> > >> scale
> > >> >>>> >> very well. I would suggest you give the asynchbase client a
> run.
> > >> >>>> >>
> > >> >>>> >> J-D
> > >> >>>> >>
> > >> >>>> >> On Tue, Jul 30, 2013 at 11:23 AM, Vladimir Rodionov
> > >> >>>> >> <[email protected]> wrote:
> > >> >>>> >> > I have been doing quite extensive testing of different read
> > >> >>>> scenarios:
> > >> >>>> >> >
> > >> >>>> >> > 1. blockcache disabled/enabled
> > >> >>>> >> > 2. data is local/remote (no good hdfs locality)
> > >> >>>> >> >
> > >> >>>> >> > and it turned out that that I can not saturate 1 RS using
> one
> > >> >>>> >> (comparable in CPU power and RAM) client host:
> > >> >>>> >> >
> > >> >>>> >> > I am running client app with 60 read threads active (with
> > >> >>>> multi-get)
> > >> >>>> >> that is going to one particular RS and
> > >> >>>> >> > this RS's load is 100 -150% (out of 3200% available) - it
> > means
> > >> >>>> that
> > >> >>>> >> load is ~5%
> > >> >>>> >> >
> > >> >>>> >> > All threads in RS are either in BLOCKED (wait) or in
> IN_NATIVE
> > >> >>>> states
> > >> >>>> >> (epoll)
> > >> >>>> >> >
> > >> >>>> >> > I attribute this to the HBase client implementation which
> > seems
> > >> >>>> to be
> > >> >>>> >> not scalable (I am going dig into client later on today).
> > >> >>>> >> >
> > >> >>>> >> > Some numbers: The maximum what I could get from Single get
> (60
> > >> >>>> threads):
> > >> >>>> >> 30K per sec. Multiget gives ~ 75K (60 threads)
> > >> >>>> >> >
> > >> >>>> >> > What are my options? I want to measure the limits and I do
> not
> > >> >>>> want to
> > >> >>>> >> run Cluster of clients against just ONE Region Server?
> > >> >>>> >> >
> > >> >>>> >> > RS config: 96GB RAM, 16(32) CPU
> > >> >>>> >> > Client : 48GB RAM 8 (16) CPU
> > >> >>>> >> >
> > >> >>>> >> > Best regards,
> > >> >>>> >> > Vladimir Rodionov
> > >> >>>> >> > Principal Platform Engineer
> > >> >>>> >> > Carrier IQ, www.carrieriq.com
> > >> >>>> >> > e-mail: [email protected]
> > >> >>>> >> >
> > >> >>>> >> >
> > >> >>>> >> > Confidentiality Notice: The information contained in this
> > >> message,
> > >> >>>> >> including any attachments hereto, may be confidential and is
> > >> >>>> intended to be
> > >> >>>> >> read only by the individual or entity to whom this message is
> > >> >>>> addressed. If
> > >> >>>> >> the reader of this message is not the intended recipient or an
> > >> agent
> > >> >>>> or
> > >> >>>> >> designee of the intended recipient, please note that any
> review,
> > >> use,
> > >> >>>> >> disclosure or distribution of this message or its attachments,
> > in
> > >> >>>> any form,
> > >> >>>> >> is strictly prohibited. If you have received this message in
> > >> error,
> > >> >>>> please
> > >> >>>> >> immediately notify the sender and/or
> > [email protected]
> > >> >>>> >> delete or destroy any copy of this message and its
> attachments.
> > >> >>>> >>
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >>
> > >> >
> > >>
> > >>
> > >
> >
>
>
