[ https://issues.apache.org/jira/browse/HBASE-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Cang reassigned HBASE-21349: ------------------------------- Assignee: Xu Cang > Cluster is going down but CatalogJanitor and Normalizer try to run and fail > noisely > ----------------------------------------------------------------------------------- > > Key: HBASE-21349 > URL: https://issues.apache.org/jira/browse/HBASE-21349 > Project: HBase > Issue Type: Bug > Affects Versions: 2.1.0 > Reporter: stack > Assignee: Xu Cang > Priority: Minor > > Shutting down can take a while. Meantime catalog janitor and or normalizer > (etc?) try to run and when they can't, they fail noisely. Looks bad: > {code} > 2018-10-19 21:23:24,962 INFO org.apache.hadoop.hbase.master.ServerManager: > Cluster shutdown set; vc1205.halxg.cloudera.com,22101,1539991730711 expired; > onlineServers=51 > 2018-10-19 21:25:54,502 WARN org.apache.hadoop.hbase.master.CatalogJanitor: > Failed scan of catalog table > java.io.IOException: connection is closed > at > org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:684) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMetaForTableRegions(MetaTableAccessor.java:679) > at > org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:185) > at > org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:137) > at > org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:243) > at > org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:116) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2018-10-19 21:25:54,507 ERROR > org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to > normalize regions. > java.io.IOException: connection is closed > at > org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690) > at > org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:240) > at > org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189) > at > org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1718) > at > org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)