[
https://issues.apache.org/jira/browse/HDDS-10651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen reassigned HDDS-10651:
---------------------------------
Assignee: Sammi Chen
> HBase WAL spitting fails due to lease recovery
> ----------------------------------------------
>
> Key: HDDS-10651
> URL: https://issues.apache.org/jira/browse/HDDS-10651
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Wei-Chiu Chuang
> Assignee: Sammi Chen
> Priority: Major
> Attachments:
> hbase-cmf-HBASE-1-REGIONSERVER-ccycloud-1.weichiu-src.root.comops.site.log.out.tgz
>
>
> HBase RS log:
> {noformat}
> 2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter:
> Splitting
> ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
> tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442,
> size=83 (83bytes)
> 2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter:
> Splitting
> ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
> tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475,
> size=83 (83bytes)
> 2024-04-04 21:33:03,340 INFO
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: Recover lease on dfs file
> ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
> ,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442
> 2024-04-04 21:33:03,340 INFO
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: Recover lease on dfs file
> ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
> ,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
> 2024-04-04 21:33:03,361 WARN
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: attempt=0 on
> file=ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1
> 712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
> after 19ms
> KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD
> org.apache.hadoop.ozone.om.exceptions.OMException: Open Key
> hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-splitting/ccycloud-2.weic
> hiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
> updated recently and is inside soft limit period
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:756)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.recoverLease(OzoneManagerProtocolClientSideTranslatorPB.java:2560)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:76)
> at com.sun.proxy.$Proxy27.recoverLease(Unknown Source)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.recoverFilePrepare(BasicRootedOzoneClientAdapterImpl.java:1384)
> at
> org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.recoverLease(RootedOzoneFileSystem.java:154)
> at
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:183)
> at
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:115)
> at
> org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:70)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:455)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitWAL(WALSplitter.java:306)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:201)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:108)
> at
> org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:100)
> at
> org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:46)
> at
> org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The Hbase code:
> https://github.com/apache/hbase/blob/e48c4485db8d9255510ee1cc9cf465e14de637d7/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java#L457
> note: HBase RS wal spliting opens an input stream to the WAL file. It
> forcefully invokes lease recovery regardless of the state of file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
