[jira] [Commented] (HBASE-20324) Hbase master fails to become active in kerberos environment
[ https://issues.apache.org/jira/browse/HBASE-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426958#comment-16426958 ] Abhishek Kulkarni commented on HBASE-20324: --- Any update on this? This is bloking us from long time. Appriciate if we have any temp steps to resolve. > Hbase master fails to become active in kerberos environment > --- > > Key: HBASE-20324 > URL: https://issues.apache.org/jira/browse/HBASE-20324 > Project: HBase > Issue Type: Bug > Environment: Hbase 2.0.0-beta2 > zookeeper-3.5.3-beta > 3 nodes Env > Kdc server on namenode > *hadoop-2.7.3* > *--Configured with keytabs(abhishekk1/2/3 are nodes)* > *abhishekk1 is namenode/hmaster* > *abhishekk2/3 are datanodes/regionservers* >Reporter: Abhishek Kulkarni >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hbase-root-master-.log, hbase-root-regionserver.log > > > > [^hbase-root-master-.log] > ^[^hbase-root-regionserver.log]^ > > ^^Trying to resolve this form last one month with different forums but not > able to resovleat all.^^ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421608#comment-16421608
]
Abhishek Kulkarni commented on HBASE-19287:
---
Can anyone update me about the mailing list?
FYI- My HDFS setup is up using same Kerberos environment.
> master hangs forever if RecoverMeta send assign meta region request to target
> server fail
> -
>
> Key: HBASE-19287
> URL: https://issues.apache.org/jira/browse/HBASE-19287
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19287-master-v3.patch,
> HBASE-19287-master-v3.patch, HBASE-19287-master-v4.patch,
> hbase-19287-master-v2.patch, master.patch
>
>
> 2017-11-10 19:26:56,019 INFO [ProcExecWrkr-1]
> procedure.RecoverMetaProcedure: pid=138,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to
> server=hadoop-slave1.hadoop,16020,1510341981454
> 2017-11-10 19:26:56,029 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454}]
> 2017-11-10 19:26:56,067 INFO [ProcExecWrkr-2]
> procedure.MasterProcedureScheduler: pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454 hbase:meta
> hbase:meta,,1.1588230740
> 2017-11-10 19:26:56,071 INFO [ProcExecWrkr-2] assignment.AssignProcedure:
> Start pid=139, ppid=138, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740,
> target=hadoop-slave1.hadoop,16020,1510341981454; rit=OFFLINE,
> location=hadoop-slave1.hadoop,16020,1510341981454; forceNewPlan=false,
> retain=false
> 2017-11-10 19:26:56,224 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
> Setting hbase:meta (replicaId=0) location in ZooKeeper as
> hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,230 INFO [ProcExecWrkr-4]
> assignment.RegionTransitionProcedure: Dispatch pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454;
> rit=OPENING, location=hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,382 INFO [ProcedureDispatcherTimeoutThread]
> procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
> serverName=hadoop-slave2.hadoop,16020,1510341988652 version=2097152
> 2017-11-10 19:26:57,542 INFO [main-EventThread]
> zookeeper.RegionServerTracker: RegionServer ephemeral node deleted,
> processing expiration [hadoop-slave2.hadoop,16020,1510341988652]
> 2017-11-10 19:26:57,543 INFO [main-EventThread] master.ServerManager: Master
> doesn't enable ServerShutdownHandler during initialization, delay expiring
> server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:58,875 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave1.hadoop,16020,1510342016106
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Triggering server recovery; existingServer
> hadoop-slave2.hadoop,16020,1510341988652 looks stale, new
> server:hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Master doesn't enable ServerShutdownHandler during
> initialization, delay expiring server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:27:49,815 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> client.RpcRetryingCallerImpl: tarted=38594 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not
> online on hadoop-slave2.hadoop,16020,1510342023184
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3290)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1370)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2401)
> at
>
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421431#comment-16421431
]
Abhishek Kulkarni commented on HBASE-19287:
---
2018-03-31 14:00:18,202 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache:
totalSize=1.03 MB, freeSize=1.38 GB, max=1.38 GB, blockCount=0, accesses=0,
hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=3239, evicted=0, evictedPerRun=0.0
2018-03-31 14:00:18,208 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache
Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0
2018-03-31 14:00:20,763 INFO [regionserver/abhishekk3:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522486816915/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522515620673
with entries=0, filesize=83 B; new WAL
/hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522486816915/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522519220738
2018-03-31 14:00:20,763 INFO [regionserver/abhishekk3:16020.logRoller]
wal.AbstractFSWAL: Archiving
hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522486816915/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522515620673
to
hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/oldWALs/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522515620673
2018-03-31 14:05:18,202 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache:
totalSize=1.03 MB, freeSize=1.38 GB, max=1.38 GB, blockCount=0, accesses=0,
hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=3269, evicted=0, evictedPerRun=0.0
> master hangs forever if RecoverMeta send assign meta region request to target
> server fail
> -
>
> Key: HBASE-19287
> URL: https://issues.apache.org/jira/browse/HBASE-19287
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19287-master-v3.patch,
> HBASE-19287-master-v3.patch, HBASE-19287-master-v4.patch,
> hbase-19287-master-v2.patch, master.patch
>
>
> 2017-11-10 19:26:56,019 INFO [ProcExecWrkr-1]
> procedure.RecoverMetaProcedure: pid=138,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to
> server=hadoop-slave1.hadoop,16020,1510341981454
> 2017-11-10 19:26:56,029 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454}]
> 2017-11-10 19:26:56,067 INFO [ProcExecWrkr-2]
> procedure.MasterProcedureScheduler: pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454 hbase:meta
> hbase:meta,,1.1588230740
> 2017-11-10 19:26:56,071 INFO [ProcExecWrkr-2] assignment.AssignProcedure:
> Start pid=139, ppid=138, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740,
> target=hadoop-slave1.hadoop,16020,1510341981454; rit=OFFLINE,
> location=hadoop-slave1.hadoop,16020,1510341981454; forceNewPlan=false,
> retain=false
> 2017-11-10 19:26:56,224 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
> Setting hbase:meta (replicaId=0) location in ZooKeeper as
> hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,230 INFO [ProcExecWrkr-4]
> assignment.RegionTransitionProcedure: Dispatch pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454;
> rit=OPENING, location=hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,382 INFO [ProcedureDispatcherTimeoutThread]
> procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
> serverName=hadoop-slave2.hadoop,16020,1510341988652 version=2097152
> 2017-11-10 19:26:57,542 INFO [main-EventThread]
> zookeeper.RegionServerTracker: RegionServer ephemeral node deleted,
> processing expiration [hadoop-slave2.hadoop,16020,1510341988652]
> 2017-11-10 19:26:57,543 INFO [main-EventThread] master.ServerManager: Master
> doesn't enable ServerShutdownHandler during initialization, delay expiring
> server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:58,875 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
>
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[ https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421429#comment-16421429 ] Abhishek Kulkarni commented on HBASE-19287: --- Dont see any error at regionserver other than warning WARN [Close-WAL-Writer-3] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522486816915/abhishekk3.pne.ven.veritas.com%2C16020%2C1522486816915.1522497620342 not finished, retry = 0 Sat Mar 31 05:00:13 EDT 2018 Starting regionserver on abhishekk3.pne.ven.veritas.com core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63395 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63395 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2018-03-31 05:00:16,504 INFO [main] regionserver.HRegionServer: STARTING executorService HRegionServer 2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: HBase 2.0.0-beta-2 2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: Source code repository git://ve0524/home/stack/hbase.git revision=9e9b347d667e1fc6165c9f8ae5ae7052147e8895 2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: Compiled by stack on Fri Mar 2 13:29:06 PST 2018 2018-03-31 05:00:16,505 INFO [main] util.VersionInfo: From source with checksum 07b3577d4d48c7a79e91a2d133e09db2 2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.tmp.dir: /tmp/hbase-root 2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.rootdir: hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase 2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true 2018-03-31 05:00:16,888 INFO [main] util.ServerCommandLine: hbase.zookeeper.quorum: abhishekk1.pne.ven.veritas.com,abhishekk2.pne.ven.veritas.com,abhishekk3.pne.ven.veritas.com 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:PATH=/home/java/jdk1.8.0_131//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/hadoop/hadoop-2.7.3//sbin:/home/hadoop/hadoop-2.7.3//bin:/home/hbase2/hbase-2.0.0-beta-2/bin 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:HBASE_PID_DIR=/var/hbase/pids 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:MAIL=/var/mail/root 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/home/hadoop/hadoop-2.7.3//lib/native 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:LOGNAME=root 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:PWD=/root 2018-03-31 05:00:16,889 INFO [main] util.ServerCommandLine: env:HADOOP_INSTALL=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:LESSOPEN=||/usr/bin/lesspipe.sh %s 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbasemaster-jaas.conf 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVERS=/home/hbase2/hbase-2.0.0-beta-2/conf/regionservers 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0 2018-03-31 05:00:16,890 INFO [main] util.ServerCommandLine: env:HBASE_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseclient-jass.conf -Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf -Dhbase.log.dir=/home/hbase2/hbase-2.0.0-beta-2/logs -Dhbase.log.file=hbase-root-regionserver-abhishekk3.pne.ven.veritas.com.log -Dhbase.home.dir=/home/hbase2/hbase-2.0.0-beta-2 -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native
[jira] [Created] (HBASE-20324) Hbase master fails to become active in kerberos environment
Abhishek Kulkarni created HBASE-20324: - Summary: Hbase master fails to become active in kerberos environment Key: HBASE-20324 URL: https://issues.apache.org/jira/browse/HBASE-20324 Project: HBase Issue Type: Bug Environment: Hbase 2.0.0-beta2 zookeeper-3.5.3-beta 3 nodes Env Kdc server on namenode *hadoop-2.7.3* *--Configured with keytabs(abhishekk1/2/3 are nodes)* *abhishekk1 is namenode/hmaster* *abhishekk2/3 are datanodes/regionservers* Reporter: Abhishek Kulkarni Attachments: hbase-root-master-.log, hbase-root-regionserver.log [^hbase-root-master-.log] ^[^hbase-root-regionserver.log]^ ^^Trying to resolve this form last one month with different forums but not able to resovleat all.^^ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[ https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421416#comment-16421416 ] Abhishek Kulkarni commented on HBASE-19287: --- 2018-03-31 05:00:27,274 WARN [RSProcedureDispatcher-pool3-t10] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915; exception=Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:27,274 INFO [RSProcedureDispatcher-pool3-t10] assignment.AssignProcedure: Retry=10 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:27,275 INFO [PEWorker-7] assignment.AssignProcedure: Retry=11 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null 2018-03-31 05:00:27,408 ERROR [PEWorker-7] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception for pid=11, state=FAILED:RECOVER_META_ASSIGN_REGIONS, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true java.lang.UnsupportedOperationException: unhandled state=RECOVER_META_ASSIGN_REGIONS at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:188) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:53) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199) at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,413 ERROR [PEWorker-7] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception for pid=11, state=FAILED:RECOVER_META_ASSIGN_REGIONS, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true java.lang.UnsupportedOperationException: unhandled state=RECOVER_META_ASSIGN_REGIONS at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:188) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:53) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199) at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,423 INFO [PEWorker-7] procedure2.ProcedureExecutor: Rolled back pid=11, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true exec-time=5.1550sec 2018-03-31 05:00:27,423 ERROR [master/abhishekk1:16000] master.HMaster: Failed to become active master org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded at org.apache.hadoop.hbase.master.assignment.AssignProcedure.startTransition(AssignProcedure.java:181) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:295) at
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[ https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421413#comment-16421413 ] Abhishek Kulkarni commented on HBASE-19287: --- at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:148) ... 25 more Caused by: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.security.SaslChallengeDecoder.tryDecodeError(SaslChallengeDecoder.java:92) at org.apache.hadoop.hbase.security.SaslChallengeDecoder.decode(SaslChallengeDecoder.java:109) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ... 14 more 2018-03-31 05:00:24,484 WARN [RSProcedureDispatcher-pool3-t1] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2018-03-31 05:00:24,485 INFO [RSProcedureDispatcher-pool3-t1] assignment.AssignProcedure: Retry=1 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,485 INFO [PEWorker-5] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:24,638 INFO [PEWorker-6] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,643 INFO [PEWorker-6] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,794 WARN [RSProcedureDispatcher-pool3-t2] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421410#comment-16421410
]
Abhishek Kulkarni commented on HBASE-19287:
---
Cant paste all log here. Also not able to attach file. Is there a way to attch
file?
Pasting few failures which are allowed.
===
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740)
2018-03-31 05:00:22,646 INFO [PEWorker-1] master.SplitLogManager: finished
splitting (more than or equal to) 0 bytes in 0 log files in
[hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting]
in 14ms
2018-03-31 05:00:22,654 INFO [PEWorker-1] procedure.RecoverMetaProcedure:
pid=11, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
failedMetaServer=null, splitWal=true; Retaining meta assignment to
server=abhishekk2.pne.ven.veritas.com,16020,1522480363659
2018-03-31 05:00:22,660 INFO [PEWorker-1] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=12, ppid=11,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659}]
2018-03-31 05:00:22,677 INFO [PEWorker-2] procedure.MasterProcedureScheduler:
pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=hbase:meta, region=1588230740,
target=abhishekk2.pne.ven.veritas.com,16020,1522480363659,
hbase:meta,,1.1588230740
2018-03-31 05:00:22,684 INFO [PEWorker-2] assignment.AssignProcedure: Starting
pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=hbase:meta, region=1588230740,
target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OFFLINE,
location=abhishekk2.pne.ven.veritas.com,16020,1522480363659;
forceNewPlan=false, retain=false
2018-03-31 05:00:22,861 INFO [master/abhishekk1:16000]
balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart
assignment.
2018-03-31 05:00:22,863 INFO [PEWorker-3] assignment.AssignProcedure: Early
suspend! pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
AssignProcedure table=hbase:meta, region=1588230740,
target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OFFLINE,
location=abhishekk2.pne.ven.veritas.com,16020,1522486814482
2018-03-31 05:00:23,629 INFO [HBase-Metrics2-1]
impl.GlobalMetricRegistriesAdapter: Registering
Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.security.access.AccessController
Metrics about HBase MasterObservers
2018-03-31 05:00:24,237 INFO [PEWorker-4] zookeeper.MetaTableLocator: Setting
hbase:meta (replicaId=0) location in ZooKeeper as
abhishekk2.pne.ven.veritas.com,16020,1522486814482
2018-03-31 05:00:24,243 INFO [PEWorker-4]
assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659;
rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482
2018-03-31 05:00:24,482 WARN [RSProcedureDispatcher-pool3-t1]
procedure.RSProcedureDispatcher: Failed dispatch to
server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0
java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020
failed on local exception: java.io.IOException:
org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
initiate failed
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at
[jira] [Comment Edited] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421195#comment-16421195
]
Abhishek Kulkarni edited comment on HBASE-19287 at 3/31/18 7:06 AM:
I still see this issue with my Hbase 2.0.0-Beta2 version. I am using Kerberos
security and not able to start Hbase Master with above error.
Zokeeper ver- zookeeper-3.5.3-beta
Log LIne-
INFO [PEWorker-2] procedure.RecoverMetaProcedure: pid=29,
state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
failedMetaServer=null, splitWal=true; Retaining meta assignment to server=
I am blocked on this, can anyone help with resolution steps?
was (Author: ahk123in):
I still see this issue with my Hbase 2.0.0-Beta2 version. I am using Kerberos
security and not able to start Hbase Master with above error.
Zokeeper ver- zookeeper-3.5.3-beta
Log LIne-
INFO [PEWorker-2] procedure.RecoverMetaProcedure: pid=29,
state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
failedMetaServer=null, splitWal=true; Retaining meta assignment to server=
> master hangs forever if RecoverMeta send assign meta region request to target
> server fail
> -
>
> Key: HBASE-19287
> URL: https://issues.apache.org/jira/browse/HBASE-19287
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19287-master-v3.patch,
> HBASE-19287-master-v3.patch, HBASE-19287-master-v4.patch,
> hbase-19287-master-v2.patch, master.patch
>
>
> 2017-11-10 19:26:56,019 INFO [ProcExecWrkr-1]
> procedure.RecoverMetaProcedure: pid=138,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to
> server=hadoop-slave1.hadoop,16020,1510341981454
> 2017-11-10 19:26:56,029 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454}]
> 2017-11-10 19:26:56,067 INFO [ProcExecWrkr-2]
> procedure.MasterProcedureScheduler: pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454 hbase:meta
> hbase:meta,,1.1588230740
> 2017-11-10 19:26:56,071 INFO [ProcExecWrkr-2] assignment.AssignProcedure:
> Start pid=139, ppid=138, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740,
> target=hadoop-slave1.hadoop,16020,1510341981454; rit=OFFLINE,
> location=hadoop-slave1.hadoop,16020,1510341981454; forceNewPlan=false,
> retain=false
> 2017-11-10 19:26:56,224 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
> Setting hbase:meta (replicaId=0) location in ZooKeeper as
> hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,230 INFO [ProcExecWrkr-4]
> assignment.RegionTransitionProcedure: Dispatch pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454;
> rit=OPENING, location=hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,382 INFO [ProcedureDispatcherTimeoutThread]
> procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
> serverName=hadoop-slave2.hadoop,16020,1510341988652 version=2097152
> 2017-11-10 19:26:57,542 INFO [main-EventThread]
> zookeeper.RegionServerTracker: RegionServer ephemeral node deleted,
> processing expiration [hadoop-slave2.hadoop,16020,1510341988652]
> 2017-11-10 19:26:57,543 INFO [main-EventThread] master.ServerManager: Master
> doesn't enable ServerShutdownHandler during initialization, delay expiring
> server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:58,875 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave1.hadoop,16020,1510342016106
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Triggering server recovery; existingServer
> hadoop-slave2.hadoop,16020,1510341988652 looks stale, new
> server:hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Master doesn't
[jira] [Commented] (HBASE-19287) master hangs forever if RecoverMeta send assign meta region request to target server fail
[
https://issues.apache.org/jira/browse/HBASE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421195#comment-16421195
]
Abhishek Kulkarni commented on HBASE-19287:
---
I still see this issue with my Hbase 2.0.0-Beta2 version. I am using Kerberos
security and not able to start Hbase Master with above error.
Zokeeper ver- zookeeper-3.5.3-beta
Log LIne-
INFO [PEWorker-2] procedure.RecoverMetaProcedure: pid=29,
state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
failedMetaServer=null, splitWal=true; Retaining meta assignment to server=
> master hangs forever if RecoverMeta send assign meta region request to target
> server fail
> -
>
> Key: HBASE-19287
> URL: https://issues.apache.org/jira/browse/HBASE-19287
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19287-master-v3.patch,
> HBASE-19287-master-v3.patch, HBASE-19287-master-v4.patch,
> hbase-19287-master-v2.patch, master.patch
>
>
> 2017-11-10 19:26:56,019 INFO [ProcExecWrkr-1]
> procedure.RecoverMetaProcedure: pid=138,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to
> server=hadoop-slave1.hadoop,16020,1510341981454
> 2017-11-10 19:26:56,029 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454}]
> 2017-11-10 19:26:56,067 INFO [ProcExecWrkr-2]
> procedure.MasterProcedureScheduler: pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454 hbase:meta
> hbase:meta,,1.1588230740
> 2017-11-10 19:26:56,071 INFO [ProcExecWrkr-2] assignment.AssignProcedure:
> Start pid=139, ppid=138, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740,
> target=hadoop-slave1.hadoop,16020,1510341981454; rit=OFFLINE,
> location=hadoop-slave1.hadoop,16020,1510341981454; forceNewPlan=false,
> retain=false
> 2017-11-10 19:26:56,224 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
> Setting hbase:meta (replicaId=0) location in ZooKeeper as
> hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,230 INFO [ProcExecWrkr-4]
> assignment.RegionTransitionProcedure: Dispatch pid=139, ppid=138,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
> region=1588230740, target=hadoop-slave1.hadoop,16020,1510341981454;
> rit=OPENING, location=hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:56,382 INFO [ProcedureDispatcherTimeoutThread]
> procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
> serverName=hadoop-slave2.hadoop,16020,1510341988652 version=2097152
> 2017-11-10 19:26:57,542 INFO [main-EventThread]
> zookeeper.RegionServerTracker: RegionServer ephemeral node deleted,
> processing expiration [hadoop-slave2.hadoop,16020,1510341988652]
> 2017-11-10 19:26:57,543 INFO [main-EventThread] master.ServerManager: Master
> doesn't enable ServerShutdownHandler during initialization, delay expiring
> server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:26:58,875 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave1.hadoop,16020,1510342016106
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Registering
> server=hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Triggering server recovery; existingServer
> hadoop-slave2.hadoop,16020,1510341988652 looks stale, new
> server:hadoop-slave2.hadoop,16020,1510342023184
> 2017-11-10 19:27:05,832 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> master.ServerManager: Master doesn't enable ServerShutdownHandler during
> initialization, delay expiring server hadoop-slave2.hadoop,16020,1510341988652
> 2017-11-10 19:27:49,815 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
> client.RpcRetryingCallerImpl: tarted=38594 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not
> online on hadoop-slave2.hadoop,16020,1510342023184
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3290)
