[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531442#comment-16531442
]
Duo Zhang commented on HBASE-20829:
-----------------------------------
[~zghaobac] FYI. Seem a problem for replaying remote wals...
{noformat}
2018-07-03 12:25:11,375 WARN [RSProcedureDispatcher-pool13-t19]
replication.SyncReplicationReplayWALRemoteProcedure(107): Replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 failed for peer id=1
org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server
asf916.gq1.ygridcore.net,33811,1530620636539 is not online
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:285)
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:276)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-03 12:25:11,440 DEBUG [Thread-2883]
replication.TestSyncReplicationStandbyKillRS(111): Server
[asf916.gq1.ygridcore.net,33811,1530620636539] marked as dead, waiting for it
to finish dead processing
2018-07-03 12:25:11,441 DEBUG [Thread-2883]
replication.TestSyncReplicationStandbyKillRS(114): Server
[asf916.gq1.ygridcore.net,33811,1530620636539] still being processed, waiting
2018-07-03 12:25:11,456 WARN [RS:3;asf916:45751] wal.AbstractFSWAL(419):
'hbase.regionserver.maxlogs' was deprecated.
2018-07-03 12:25:11,457 INFO [RS:3;asf916:45751] wal.AbstractFSWAL(424): WAL
configuration: blocksize=256 MB, rollsize=128 MB,
prefix=asf916.gq1.ygridcore.net%2C45751%2C1530620709275, suffix=,
logDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/WALs/asf916.gq1.ygridcore.net,45751,1530620709275,
archiveDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/oldWALs
2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-4]
asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId
=
DatanodeInfoWithStorage[127.0.0.1:38997,DS-6002160d-388b-4840-8538-e4c2255108be,DISK]
2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-5]
asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId
=
DatanodeInfoWithStorage[127.0.0.1:45904,DS-e189e3c8-a1bd-475c-86c0-3891e541fc6e,DISK]
2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-3]
asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId
=
DatanodeInfoWithStorage[127.0.0.1:39589,DS-62ced3f8-35c4-4904-80cc-4d514b8f4050,DISK]
2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
procedure2.ProcedureExecutor(887): Stored pid=30,
state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
server=asf916.gq1.ygridcore.net,33811,1530620636539, splitWal=true, meta=true
2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
assignment.AssignmentManager(1321):
Added=asf916.gq1.ygridcore.net,33811,1530620636539 to dead servers, submitted
shutdown handler to be executed meta=true
2018-07-03 12:25:11,498 INFO [PEWorker-6] procedure.ServerCrashProcedure(118):
Start pid=30, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
server=asf916.gq1.ygridcore.net,33811,1530620636539, splitWal=true, meta=true
2018-07-03 12:25:11,500 WARN [RegionServerTracker-0]
replication.SyncReplicationReplayWALRemoteProcedure(107): Replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 failed for peer id=1
org.apache.hadoop.hbase.DoNotRetryIOException: server not online
asf916.gq1.ygridcore.net,33811,1530620636539
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher.abortPendingOperations(RSProcedureDispatcher.java:130)
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher.abortPendingOperations(RSProcedureDispatcher.java:60)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher$BufferNode.abortOperationsInQueue(RemoteProcedureDispatcher.java:380)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.removeNode(RemoteProcedureDispatcher.java:193)
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher.serverRemoved(RSProcedureDispatcher.java:143)
at
org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:610)
at
org.apache.hadoop.hbase.master.RegionServerTracker.refresh(RegionServerTracker.java:160)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-03 12:25:11,503 WARN [PEWorker-4]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,503 WARN [PEWorker-4]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,503 WARN [PEWorker-4]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,503 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,504 WARN [PEWorker-7]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,505 WARN [PEWorker-11]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,505 WARN [PEWorker-8]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,505 WARN [PEWorker-8]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
2018-07-03 12:25:11,505 WARN [PEWorker-8]
replication.SyncReplicationReplayWALRemoteProcedure(162): Can not add remote
operation for replay wals
[remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
on asf916.gq1.ygridcore.net,33811,1530620636539 for peer id=1, this usually
because the server is already dead, retry
{noformat}
> Remove the addFront assertion in MasterProcedureScheduler.doAdd
> ---------------------------------------------------------------
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829-v1.patch,
> HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
