[jira] [Comment Edited] (HBASE-21519) Namespace region is never assigned in a HM failover scenario when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917440#comment-16917440 ] Pankaj Kumar edited comment on HBASE-21519 at 8/28/19 5:55 AM: --- HBASE-21843 solves this problem. [~stack] sir, should we add the test case? Kindly provide your opinion. was (Author: pankaj2461): HBASE-21843 solves this problem. Resolving it as duplicate. > Namespace region is never assigned in a HM failover scenario when multiwal is > enabled > - > > Key: HBASE-21519 > URL: https://issues.apache.org/jira/browse/HBASE-21519 > Project: HBase > Issue Type: Bug > Components: master, wal >Affects Versions: 2.1.1 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Attachments: HBASE-21519.branch-2.patch, providerId_value.png > > > In our test env we found that namespace region is never be assigned on HM > failover scenario when multiwal feature is enabled, > {noformat} > 2018-11-28 01:38:28,085 WARN [master/HM-1:16000:becomeActiveMaster] > master.HMaster: > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. is NOT > online; state=\{31f6d3383af09e18e1e81ca02a93de15 state=OPEN, > ts=1543340156928, server=RS-2,16020,1543339824397}; > ServerCrashProcedures=false. Master startup cannot progress, in > holding-pattern until region onlined. > {noformat} > And finally HM abort with following error, > {noformat} > 2018-11-28 01:39:16,858 ERROR > [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Master > failed to complete initialization after 24ms. Please consider submitting > a bug report including a thread dump of this process. > 2018-11-28 01:39:18,980 ERROR > [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Zombie > Master exiting. Thread dump to stdout > {noformat} > Stack trace: > {noformat} > Thread 102 (master/HM-1:16000:becomeActiveMaster): > State: TIMED_WAITING > Blocked count: 100 > Waited count: 246 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:148) > org.apache.hadoop.hbase.master.HMaster.isRegionOnline(HMaster.java:1166) > > org.apache.hadoop.hbase.master.HMaster.waitForNamespaceOnline(HMaster.java:1187) > > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1044) > > org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2285) > org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:590) > org.apache.hadoop.hbase.master.HMaster$$Lambda$40/1078246575.run(Unknown > Source) > java.lang.Thread.run(Thread.java:745) > {noformat} > > Step to reproduce: > 1) Setup a HBase cluster with 1/2 HM (say HM-1) and 2 RS(say RS-1 & RS-2) > 2) Enable multiwal feature with following configuration setting and start > the cluster, > {noformat} > > hbase.wal.provider > multiwal > > > hbase.wal.regiongrouping.strategy > identity > > {noformat} > 3) Make sure meta and namespace regions are assigned on different RS, suppose > RS-1 & RS-2 respectively. > 4) Create table 't1' > 5) Flush the meta table explicitly > 6) Kill the RS-2, so during RS-2 SCP all regions including namespace region > will be assigned to RS-1. > 7) Now Kill RS-1 before meta flush happen. Here both RS-2 & RS-1 are > shutdown now. > 8) Stop the HM and start RS-1 & RS-2. > 9) Now start the HM. > Meta region is assigned successfully but HM is keep waiting for the namespace > region onlline (Master startup cannot progress, in holding-pattern until > region onlined) and abort with timeout. > Observation: > 1) After step-3 namespace region was assigned to RS-2 and meta entry was as > follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339860920, value=RS-2:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode, timestamp=1543339860920, value=1543339824397 > {noformat} > 2) After step-6 namespace region was assigned to RS-1 and meta entry was as > follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339880920, value=RS-1:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode, timestamp=1543339880920, value=1543339829288 > {noformat} > 3) After Step-9, meta entry for namespace region was as follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339860920, value=RS-2:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode,
[jira] [Comment Edited] (HBASE-21519) Namespace region is never assigned in a HM failover scenario when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917440#comment-16917440 ] Pankaj Kumar edited comment on HBASE-21519 at 8/28/19 5:50 AM: --- HBASE-21843 solves this problem. Resolving it as duplicate. was (Author: pankaj2461): HBASE-21843 solves this problem. > Namespace region is never assigned in a HM failover scenario when multiwal is > enabled > - > > Key: HBASE-21519 > URL: https://issues.apache.org/jira/browse/HBASE-21519 > Project: HBase > Issue Type: Bug > Components: master, wal >Affects Versions: 2.1.1 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Attachments: HBASE-21519.branch-2.patch, providerId_value.png > > > In our test env we found that namespace region is never be assigned on HM > failover scenario when multiwal feature is enabled, > {noformat} > 2018-11-28 01:38:28,085 WARN [master/HM-1:16000:becomeActiveMaster] > master.HMaster: > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. is NOT > online; state=\{31f6d3383af09e18e1e81ca02a93de15 state=OPEN, > ts=1543340156928, server=RS-2,16020,1543339824397}; > ServerCrashProcedures=false. Master startup cannot progress, in > holding-pattern until region onlined. > {noformat} > And finally HM abort with following error, > {noformat} > 2018-11-28 01:39:16,858 ERROR > [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Master > failed to complete initialization after 24ms. Please consider submitting > a bug report including a thread dump of this process. > 2018-11-28 01:39:18,980 ERROR > [ActiveMasterInitializationMonitor-1543338648565] master.HMaster: Zombie > Master exiting. Thread dump to stdout > {noformat} > Stack trace: > {noformat} > Thread 102 (master/HM-1:16000:becomeActiveMaster): > State: TIMED_WAITING > Blocked count: 100 > Waited count: 246 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:148) > org.apache.hadoop.hbase.master.HMaster.isRegionOnline(HMaster.java:1166) > > org.apache.hadoop.hbase.master.HMaster.waitForNamespaceOnline(HMaster.java:1187) > > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1044) > > org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2285) > org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:590) > org.apache.hadoop.hbase.master.HMaster$$Lambda$40/1078246575.run(Unknown > Source) > java.lang.Thread.run(Thread.java:745) > {noformat} > > Step to reproduce: > 1) Setup a HBase cluster with 1/2 HM (say HM-1) and 2 RS(say RS-1 & RS-2) > 2) Enable multiwal feature with following configuration setting and start > the cluster, > {noformat} > > hbase.wal.provider > multiwal > > > hbase.wal.regiongrouping.strategy > identity > > {noformat} > 3) Make sure meta and namespace regions are assigned on different RS, suppose > RS-1 & RS-2 respectively. > 4) Create table 't1' > 5) Flush the meta table explicitly > 6) Kill the RS-2, so during RS-2 SCP all regions including namespace region > will be assigned to RS-1. > 7) Now Kill RS-1 before meta flush happen. Here both RS-2 & RS-1 are > shutdown now. > 8) Stop the HM and start RS-1 & RS-2. > 9) Now start the HM. > Meta region is assigned successfully but HM is keep waiting for the namespace > region onlline (Master startup cannot progress, in holding-pattern until > region onlined) and abort with timeout. > Observation: > 1) After step-3 namespace region was assigned to RS-2 and meta entry was as > follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339860920, value=RS-2:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode, timestamp=1543339860920, value=1543339824397 > {noformat} > 2) After step-6 namespace region was assigned to RS-1 and meta entry was as > follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339880920, value=RS-1:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode, timestamp=1543339880920, value=1543339829288 > {noformat} > 3) After Step-9, meta entry for namespace region was as follows, > {noformat} > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:server, timestamp=1543339860920, value=RS-2:16020 > hbase:namespace,,1543339859614.31f6d3383af09e18e1e81ca02a93de15. > column=info:serverstartcode, timestamp=1543339860920, value=1543339824397 > {noformat} > During SCP we