After some more digging, I discovered that during the time that the RS is stuck
the kernel message buffer outputted only this message
"[1031214.108110] XFS: java(6522) possible memory allocation deadlock size
32944 in kmem_alloc (mode:0x2400240)"
From my reading online, the cause of this error appears to generally be
excessive memory and file fragmentation. We haven't changed the mslab config
and we are running HBase 1.3.0 so it should be running by default. The issue
tends to arise consistently and regularly (every 10 or so days) and once one
node is affected other nodes start to follow after a few hours. What could be
causing this to happen and is there any way to prevent or minimize
fragmentation?
Best,
Srinidhi
On 3/29/19, 11:02 AM, "Srinidhi Muppalla" <srinid...@trulia.com> wrote:
Stack and Ram,
Attached the thread dumps. 'Jstack normal' is the normal node. 'Jstack
problematic' was taken when the node was stuck.
We don't have full I/O stats for the problematic node. Unfortunately, it
was impacting production so we had to recreate the cluster as soon as possible
and couldn't get full data. I attached the dashboards with the wait I/O and
other CPU stats. Thanks for helping look into the issue!
Best,
Srinidhi
On 3/28/19, 2:41 PM, "Stack" <st...@duboce.net> wrote:
Mind putting up a thread dump?
How many spindles?
If you compare the i/o stats between a good RS and a stuck one, how do
they
compare?
Thanks,
S
On Wed, Mar 27, 2019 at 11:57 AM Srinidhi Muppalla
<srinid...@trulia.com>
wrote:
> Hello,
>
> We've noticed an issue in our HBase cluster where one of the
> region-servers has a spike in I/O wait associated with a spike in
Load for
> that node. As a result, our request times to the cluster increase
> dramatically. Initially, we suspected that we were experiencing
> hotspotting, but even after temporarily blocking requests to the
highest
> volume regions on that region-servers the issue persisted. Moreover,
when
> looking at request counts to the regions on the region-server from the
> HBase UI, they were not particularly high and our own application
level
> metrics on the requests we were making were not very high either. From
> looking at a thread dump of the region-server, it appears that our
get and
> scan requests are getting stuck when trying to read from the blocks
in our
> bucket cache leaving the threads in a 'runnable' state. For context,
we are
> running HBase 1.30 on a cluster backed by S3 running on EMR and our
bucket
> cache is running in File mode. Our region-servers all have SSDs. We
have a
> combined cache with the L1 standard LRU cache and the L2 file mode
bucket
> cache. Our Bucket Cache utilization is less than 50% of the allocated
space.
>
> We suspect that part of the issue is our disk space utilization on the
> region-server as our max disk space utilization also increased as this
> happened. What things can we do to minimize disk space utilization?
The
> actual HFiles are on S3 -- only the cache, application logs, and write
> ahead logs are on the region-servers. Other than the disk space
> utilization, what factors could cause high I/O wait in HBase and is
there
> anything we can do to minimize it?
>
> Right now, the only thing that works is terminating and recreating the
> cluster (which we can do safely because it's S3 backed).
>
> Thanks!
> Srinidhi
>
