Zheng Hu created HBASE-20531: -------------------------------- Summary: RS may throw NPE when close meta regions in shutdown procedure. Key: HBASE-20531 URL: https://issues.apache.org/jira/browse/HBASE-20531 Project: HBase Issue Type: Bug Reporter: Zheng Hu Assignee: Zheng Hu
See also : https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322 The NPE stack is as following: {code} 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception while closing region tes t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close ***** java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709) at org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718) at org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649) at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929) at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more {code} In HRegionServer#run(), we have the following: {code} @Override public void run() { ... // Stop the quota manager if (rsQuotaManager != null) { rsQuotaManager.stop(); } if (rsSpaceQuotaManager != null) { rsSpaceQuotaManager.stop(); rsSpaceQuotaManager = null; } ... // Closing the compactSplit thread before closing meta regions if (!this.killed && containsMetaTableRegions()) { if (!abortRequested || this.fsOk) { if (this.compactSplitThread != null) { this.compactSplitThread.join(); this.compactSplitThread = null; } closeMetaTableRegions(abortRequested); } } } {code} We stop the rsSpaceQuotaManager firstly, and then close the meta region, but when close meta region, we need to use rsSpaceQuotaManager to reportFileArchivalForQuotas() , just as the stack trace said ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)