Re: Hot Region Server With No Hot Region

On Tue, Dec 13, 2016 at 12:47 PM, Saad Mufti wrote: > Thanks everyone for the feedback. We tracked this down to having a bad > design using dynamic columns, there were a few (very few) rows that > accumulated up to 200,000 dynamic columns. When we got any activity that >

Re: Hot Region Server With No Hot Region

I was looking at CellCounter but it doesn't provide what you are looking for. Maybe we can enhance it such that given threshold on the number of qualifiers in a row (say 100,000), output the rows which have at least these many qualifiers. On Tue, Dec 13, 2016 at 12:47 PM, Saad Mufti

Re: Hot Region Server With No Hot Region

Thanks everyone for the feedback. We tracked this down to having a bad design using dynamic columns, there were a few (very few) rows that accumulated up to 200,000 dynamic columns. When we got any activity that caused us to try to read one of these rows, it resulted in a hot region server.

Re: Hot Region Server With No Hot Region

I took a look at the stack trace. Region server log would give us more detail on the frequency and duration of compactions. Cheers On Sat, Dec 3, 2016 at 7:39 AM, Jeremy Carroll wrote: > I would check compaction, investigate throttling if it's causing high CPU. > > On

Re: Hot Region Server With No Hot Region

I would check compaction, investigate throttling if it's causing high CPU. On Sat, Dec 3, 2016 at 6:20 AM Saad Mufti wrote: > No. > > > Saad > > > On Fri, Dec 2, 2016 at 3:27 PM, Ted Yu wrote: > > > Some how I couldn't access the pastebin (I

Re: Hot Region Server With No Hot Region

No. Saad On Fri, Dec 2, 2016 at 3:27 PM, Ted Yu wrote: > Some how I couldn't access the pastebin (I am in China now). > Did the region server showing hotspot host meta ? > Thanks > > On Friday, December 2, 2016 11:53 AM, Saad Mufti >

Re: Hot Region Server With No Hot Region

Some how I couldn't access the pastebin (I am in China now). Did the region server showing hotspot host meta ? Thanks On Friday, December 2, 2016 11:53 AM, Saad Mufti wrote: We're in AWS with D2.4xLarge instances. Each instance has 12 independent spindles/disks

Re: Hot Region Server With No Hot Region

We're in AWS with D2.4xLarge instances. Each instance has 12 independent spindles/disks from what I can tell. We have charted get_rate and mutate_rate by host and a) mutate_rate shows no real outliers b) read_rate shows the overall rate on the "hotspot" region server is a bit higher than every

Re: Hot Region Server With No Hot Region

Here is what I see... * Short Compaction Running on Heap "regionserver/ip-10-99-181-146.aolp-prd.us-east-1.ec2.aolcloud.net/10.99.181.146:60020-shortCompactions-1480229281547" - Thread t@242 java.lang.Thread.State: RUNNABLE at

Re: Hot Region Server With No Hot Region

Hi Ted, Finally we have another hotspot going on, same symptoms as before, here is the pastebin for the stack trace from the region server that I obtained via VisualVM: http://pastebin.com/qbXPPrXk Would really appreciate any insight you or anyone else can provide. Thanks. Saad On Thu,

Re: Hot Region Server With No Hot Region

Sure will, the next time it happens. Thanks!!! Saad On Thu, Dec 1, 2016 at 5:01 PM, Ted Yu wrote: > From #2 in the initial email, the hbase:meta might not be the cause for > the hotspot. > > Saad: > Can you pastebin stack trace of the hot region server when

Re: Hot Region Server With No Hot Region

From #2 in the initial email, the hbase:meta might not be the cause for the hotspot. Saad: Can you pastebin stack trace of the hot region server when this happens again ? Thanks > On Dec 2, 2016, at 4:48 AM, Saad Mufti wrote: > > We used a pre-split into 1024 regions

Re: Hot Region Server With No Hot Region

We used a pre-split into 1024 regions at the start but we miscalculated our data size, so there were still auto-splits storms at the beginning as data size stabilized, it has ended up at around 9500 or so regions, plus a few thousand regions for a few other tables (much smaller). But haven't had

Re: Hot Region Server With No Hot Region

Saad, Region move or split causes client connections to simultaneously refresh their meta. Key word is supposed. We have seen meta hot spotting from time to time and on different versions at Splice Machine. How confident are you in your hashing algorithm? Regards, John Leach > On Dec

Re: Hot Region Server With No Hot Region

No never thought about that. I just figured out how to locate the server for that table after you mentioned it. We'll have to keep an eye on it next time we have a hotspot to see if it coincides with the hotspot server. What would be the theory for how it could become a hotspot? Isn't the client

Re: Hot Region Server With No Hot Region

Saad, Did you validate that Meta is not on the “Hot” region server? Regards, John Leach > On Dec 1, 2016, at 1:50 PM, Saad Mufti wrote: > > Hi, > > We are using HBase 1.0 on CDH 5.5.2 . We have taken great care to avoid > hotspotting due to inadvertent data patterns