[ https://issues.apache.org/jira/browse/HBASE-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661717#comment-16661717 ]
Hudson commented on HBASE-21349: -------------------------------- Results for branch branch-2.1 [build #522 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522/]: (/) *{color:green}+1 overall{color}* ---- details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//JDK8_Nightly_Build_Report_(Hadoop2)/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/522//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Cluster is going down but CatalogJanitor and Normalizer try to run and fail > noisely > ----------------------------------------------------------------------------------- > > Key: HBASE-21349 > URL: https://issues.apache.org/jira/browse/HBASE-21349 > Project: HBase > Issue Type: Bug > Affects Versions: 2.1.0 > Reporter: stack > Assignee: Xu Cang > Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21349.master.002.patch, > HBASE-21349.master.002.patch, HBASE-21349.master.002.patch, > HBASE-22349.master.001.patch > > > Shutting down can take a while. Meantime catalog janitor and or normalizer > (etc?) try to run and when they can't, they fail noisely. Looks bad: > {code} > 2018-10-19 21:23:24,962 INFO org.apache.hadoop.hbase.master.ServerManager: > Cluster shutdown set; vc1205.halxg.cloudera.com,22101,1539991730711 expired; > onlineServers=51 > 2018-10-19 21:25:54,502 WARN org.apache.hadoop.hbase.master.CatalogJanitor: > Failed scan of catalog table > java.io.IOException: connection is closed > at > org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:684) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMetaForTableRegions(MetaTableAccessor.java:679) > at > org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:185) > at > org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:137) > at > org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:243) > at > org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:116) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2018-10-19 21:25:54,507 ERROR > org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to > normalize regions. > java.io.IOException: connection is closed > at > org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734) > at > org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690) > at > org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:240) > at > org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189) > at > org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1718) > at > org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)