Pratyush Bhatt created HDDS-8199:
------------------------------------
Summary: "Directory is not empty" while running Hbase(configured
to run on Ozone) LoadTestTool
Key: HDDS-8199
URL: https://issues.apache.org/jira/browse/HDDS-8199
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Filesystem
Affects Versions: 1.3.0
Reporter: Pratyush Bhatt
After running:
{noformat}
hbase org.apache.hadoop.hbase.util.LoadTestTool -start_key 0
-num_keys 100000 -bloom ROWCOL -compression NONE -write 10:100:100 -read 100:20
-tn loadtest_d1{noformat}
the command fails after being in execution state for nearly 2.5 Hours. There
were many exceptions like below:
{noformat}
2023-03-15 20:11:12,660|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|
2023-03-15 20:11:12,660|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:142)
2023-03-15 20:11:12,661|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.client.HTable.put(HTable.java:514)
2023-03-15 20:11:12,661|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:144)
2023-03-15 20:11:12,661|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:117)
2023-03-15 20:11:12,662|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Caused by:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to
address=ozpc4-cf4-2.ozpc4-cf4.root.hwx.site/172.27.27.133:22101 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call[id=162625,methodName=Mutate], waitTime=60011ms, rpcTimeout=60000ms
2023-03-15 20:11:12,662|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:223)
2023-03-15 20:11:12,662|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:384)
2023-03-15 20:11:12,662|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88)
2023-03-15 20:11:12,663|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:415)
2023-03-15 20:11:12,663|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:411)
2023-03-15 20:11:12,663|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:107)
2023-03-15 20:11:12,663|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:136)
2023-03-15 20:11:12,664|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
2023-03-15 20:11:12,664|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hbase.thirdparty.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
2023-03-15 20:11:12,664|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
2023-03-15 20:11:12,664|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
2023-03-15 20:11:12,665|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
2023-03-15 20:11:12,665|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
java.lang.Thread.run(Thread.java:750)
2023-03-15 20:11:12,665|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Caused by:
org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call[id=162625,methodName=Mutate], waitTime=60011ms, rpcTimeout=60000ms
2023-03-15 20:11:12,665|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|at
org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:137)
2023-03-15 20:11:12,666|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|... 6 more
2023-03-15 20:11:12,666|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|{noformat}
But still the keys were getting written(very slowly tho, for HBase on HDFS,
this command takes approx 3-4 minutes), towards the end the key write starts
failing:
{noformat}
2023-03-15 22:30:21,612|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|23/03/15 22:30:21 INFO
util.MultiThreadedAction: [W:101] Keys=99680, cols=1.1 M, time=02:37:21
Overall: [keys/s= 10, latency=4268.13 ms] Current: [keys/s=47, latency=1949.67
ms], wroteUpTo=99179, wroteQSize=400
2023-03-15 22:30:26,566|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|23/03/15 22:30:26 INFO
util.MultiThreadedAction: [R:20] Keys=7254451, cols=86.6 M, time=02:37:26
Overall: [keys/s= 767, latency=20.86 ms] Current: [keys/s=964, latency=23.55
ms], verified=7254451, READ FAILURES=66
2023-03-15 22:30:26,608|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|23/03/15 22:30:26 INFO
util.MultiThreadedAction: [W:82] Keys=100000, cols=1.1 M, time=02:37:26
Overall: [keys/s= 10, latency=4259.79 ms] Current: [keys/s=64, latency=1662.05
ms], wroteUpTo=99770, wroteQSize=148
2023-03-15 22:30:29,978|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write keys: 151
2023-03-15 22:30:30,027|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13207
2023-03-15 22:30:30,072|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13223
2023-03-15 22:30:30,114|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13240
2023-03-15 22:30:30,159|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13243
2023-03-15 22:30:30,204|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13255
2023-03-15 22:30:30,252|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13256
2023-03-15 22:30:30,296|INFO|MainThread|machine.py:203 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key: 13260
2023-03-15 22:30:30,341|INFO|MainThread|machine.py:210 -
run()||GUID=12e6a591-a42f-4d02-937a-bdc9d62acd1f|Failed to write key:
13271{noformat}
Checked the Master log during that time, was getting
org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl:
{noformat}
2023-03-15 22:30:43,417 INFO org.apache.hadoop.hbase.master.SplitLogManager:
ofs://ozone1/vol1/bucket1/hbase/WALs/ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027-splitting
dir is empty, no logs to split.
2023-03-15 22:30:43,417 INFO org.apache.hadoop.hbase.master.SplitWALManager:
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027 WAL count=0, meta=false
2023-03-15 22:30:43,449 ERROR
org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl: delete key failed
Directory is not empty.
Key:hbase/WALs/ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027-splitting
2023-03-15 22:30:43,449 WARN org.apache.hadoop.hbase.master.MasterWalManager:
Failed archiving meta log for server
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027
org.apache.hadoop.fs.PathIsNotEmptyDirectoryException: `Directory is not empty.
Key:hbase/WALs/ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027-splitting':
Directory is not empty
at
org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.deleteObject(BasicRootedOzoneClientAdapterImpl.java:548)
at
org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.delete(BasicRootedOzoneFileSystem.java:646)
at
org.apache.hadoop.hbase.master.MasterWalManager.archiveMetaLog(MasterWalManager.java:395)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.cleanupSplitDir(ServerCrashProcedure.java:280)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:225)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:936)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,473 ERROR
org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl: delete key failed
Directory is not empty.
Key:hbase/WALs/ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027-splitting
2023-03-15 22:30:43,476 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception: pid=419, state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR,
locked=true; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false
java.lang.NullPointerException
at java.util.Arrays.stream(Arrays.java:5004)
at
org.apache.hadoop.hbase.master.SplitWALManager.deleteWALDir(SplitWALManager.java:132)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.cleanupSplitDir(ServerCrashProcedure.java:282)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:225)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:936)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,534 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception for pid=419, state=FAILED:SERVER_CRASH_SPLIT_LOGS,
locked=true, exception=java.lang.NullPointerException via CODE-BUG: Uncaught
runtime exception: pid=419, state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR,
locked=true; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false:java.lang.NullPointerException; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false
java.lang.UnsupportedOperationException: unhandled
state=SERVER_CRASH_DELETE_SPLIT_WALS_DIR
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:378)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:212)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:952)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1569)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1502)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1353)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,547 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception for pid=419, state=FAILED:SERVER_CRASH_GET_REGIONS,
locked=true, exception=java.lang.NullPointerException via CODE-BUG: Uncaught
runtime exception: pid=419, state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR,
locked=true; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false:java.lang.NullPointerException; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false
java.lang.UnsupportedOperationException: unhandled state=SERVER_CRASH_SPLIT_LOGS
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:378)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:212)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:952)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1569)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1502)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1353)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,578 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception for pid=419, state=FAILED:SERVER_CRASH_START, locked=true,
exception=java.lang.NullPointerException via CODE-BUG: Uncaught runtime
exception: pid=419, state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR,
locked=true; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false:java.lang.NullPointerException; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false
java.lang.UnsupportedOperationException: unhandled
state=SERVER_CRASH_GET_REGIONS
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:378)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:212)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:952)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1569)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1502)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1353)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,592 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception for pid=419, state=FAILED:SERVER_CRASH_START, locked=true,
exception=java.lang.NullPointerException via CODE-BUG: Uncaught runtime
exception: pid=419, state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR,
locked=true; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false:java.lang.NullPointerException; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false
java.lang.UnsupportedOperationException: unhandled state=SERVER_CRASH_START
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:378)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:65)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:212)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:952)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1569)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1502)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1353)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1968)
2023-03-15 22:30:43,607 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rolled back pid=419,
state=ROLLEDBACK, exception=java.lang.NullPointerException via CODE-BUG:
Uncaught runtime exception: pid=419,
state=RUNNABLE:SERVER_CRASH_DELETE_SPLIT_WALS_DIR, locked=true;
ServerCrashProcedure ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027,
splitWal=true, meta=false:java.lang.NullPointerException; ServerCrashProcedure
ozpc4-cf4-6.ozpc4-cf4.root.hwx.site,22101,1678906419027, splitWal=true,
meta=false exec-time=297 msec
2023-03-15 22:31:02,422 INFO
org.apache.hadoop.hbase.master.RegionServerTracker: RegionServer ephemeral node
deleted, processing expiration
[ozpc4-cf4-4.ozpc4-cf4.root.hwx.site,22101,1678906419528]
2023-03-15 22:31:02,424 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing expiration of
ozpc4-cf4-4.ozpc4-cf4.root.hwx.site,22101,1678906419528 on
ozpc4-cf4-1.ozpc4-cf4.root.hwx.site,22001,1678906420490{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
