Hi Ram, For this test, the data is synthetically generated and the keys are just random fixed-width integers. We're loading into a single table with a one column family. The real data would be less uniform, but we just want to get an idea of whether or not it is feasible.
- Amit On Wed, Nov 16, 2011 at 9:07 PM, Ramkrishna S Vasudevan < [email protected]> wrote: > > Hi Amit > > As you said the regions may be distributed evenly across RS, if you can see > if the puts are reaching to a particular RS only at any point of time it > will surely overload the RS. > > As Stack pointed out, what is your schema and how is your row key designed > ? > > Regards > Ram > > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Stack > Sent: Thursday, November 17, 2011 9:29 AM > To: [email protected] > Cc: lars hofhansl > Subject: Re: Help with continuous loading configuration > > On Wed, Nov 16, 2011 at 4:09 PM, Amit Jain <[email protected]> wrote: > > On Wed, Nov 16, 2011 at 3:35 PM, Stack <[email protected]> wrote: > > > >> On Wed, Nov 16, 2011 at 3:26 PM, Amit Jain <[email protected]> wrote: > >> > Hi Lars, > >> > > >> > The keys are arriving in random order. The HBase monitoring page > shows > >> > evenly distributed load across all of the region servers. > >> > >> What kind of ops rates are you seeing? They are running nice and > >> smooth across all servers? No stuttering? Whats your regionserver > >> logs look like? > >> > >> Are you presplitting your table or just letting hbase run and do up the > >> splits? > >> > > > > As far as I can tell, the operations look smooth across all servers. > We're > > not doing any pre-splitting, just letting HBase do the splits. > > > > So, how many requests per second per server. > > How many column families? What size are the puts on average? > > > > Well, it looks like half of the regions are in the 25-32 file range and > the > > other half just have 1 or 2 files. This was when we ran it with a > > compactionThreshold of 15. > > > > So, its this count even after the load comes off? Maybe compactions > get a chance to cut in and it should shrink them. > > > > How can I tell by looking at the region server logs if we're seeing a > "high > > write rate" ? > > Look at UI for basic ops/second. > > > > I have read through that section of the HBase book. There is plenty of > CPU > > available. How do I up the number of concurrent handlers? Increase > > hbase.regionserver.handler.count ? > > > > Yes. You have it pretty low at the moment. > > What kinda of performance are you looking for? > > Post your configs so we can look at them. Post a bit of your > regionserver log and your table schema. > St.Ack > >
