[
https://issues.apache.org/jira/browse/HBASE-24469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Javier Akira Luca de Tena updated HBASE-24469:
----------------------------------------------
Description:
Found out that after an ungraceful Datanode shutdown, the number of HBase
active handlers started to grow, making RegionServer stuck and unusable.
Took the thread dump and found out multiple read handlers were in some kind of
dead lock state and also write handlers stuck.
This also caused to not be able to flush the memstore because it was waiting
for this lock:
[https://github.com/apache/hbase/blob/136414dd72a80f379b80cd6f74b5b6ebd78f33ec/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1225]
Without being able to flush it, I could not gracefully stop the RegionServer,
since we can't move out the flushing region.
Found out that the real issue was in Hadoop's DFSInputStream. When no hedged
reads succeed, the internal hedgedService.take() call hangs forever since it's
internally using a BlockingQueue:
[https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1435]
This is the Hadoop side issue: https://issues.apache.org/jira/browse/HDFS-11303
and it's fixed for 2.9.0.
This is not directly related with HBase code, but just wanted community to be
aware that with current used Hadoop used version (2.8.5), this issue could
happen.
Reproduced in HBase 1.4.13, but I think it affects to other versions.
I would like to suggest to upgrade the used Hadoop version to 2.9.0.
was:
Found out that after an ungraceful Datanode shutdown, the number of HBase
active handlers started to grow, making RegionServer stuck and unusable.
Took the thread dump and found out multiple read handlers were in some kind of
dead lock state and also write handlers stuck.
This also caused to not be able to flush the memstore because it was waiting
for this lock:
[https://github.com/apache/hbase/blob/136414dd72a80f379b80cd6f74b5b6ebd78f33ec/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1225]
Without being able to flush it, I could not gracefully stop the RegionServer,
since we can't move out the flushing region.
Found out that the real issue was in Hadoop's DFSInputStream. When no hedged
reads succeed, the internal hedgedService.take() call hangs forever since it's
internally using a BlockingQueue:
[https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1435]
This is the Hadoop side issue: https://issues.apache.org/jira/browse/HDFS-11303
and it's fixed for 2.9.0.
This is not directly related with HBase code, but just wanted community to be
aware that with current used Hadoop used version (2.8.5), this issue could
happen.
I would like to suggest to upgrade the used Hadoop version to 2.9.0.
> Hedged read might hang infinitely if read data from all DN failed
> -----------------------------------------------------------------
>
> Key: HBASE-24469
> URL: https://issues.apache.org/jira/browse/HBASE-24469
> Project: HBase
> Issue Type: Bug
> Reporter: Javier Akira Luca de Tena
> Priority: Critical
>
> Found out that after an ungraceful Datanode shutdown, the number of HBase
> active handlers started to grow, making RegionServer stuck and unusable.
> Took the thread dump and found out multiple read handlers were in some kind
> of dead lock state and also write handlers stuck.
> This also caused to not be able to flush the memstore because it was waiting
> for this lock:
> [https://github.com/apache/hbase/blob/136414dd72a80f379b80cd6f74b5b6ebd78f33ec/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1225]
> Without being able to flush it, I could not gracefully stop the RegionServer,
> since we can't move out the flushing region.
>
> Found out that the real issue was in Hadoop's DFSInputStream. When no hedged
> reads succeed, the internal hedgedService.take() call hangs forever since
> it's internally using a BlockingQueue:
> [https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1435]
>
> This is the Hadoop side issue:
> https://issues.apache.org/jira/browse/HDFS-11303 and it's fixed for 2.9.0.
> This is not directly related with HBase code, but just wanted community to be
> aware that with current used Hadoop used version (2.8.5), this issue could
> happen.
>
> Reproduced in HBase 1.4.13, but I think it affects to other versions.
> I would like to suggest to upgrade the used Hadoop version to 2.9.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
