Junhong Xu created HBASE-25205:
----------------------------------
Summary: Corrupted hfiles append timestamp every time the region
is trying to open
Key: HBASE-25205
URL: https://issues.apache.org/jira/browse/HBASE-25205
Project: HBase
Issue Type: Bug
Reporter: Junhong Xu
Assignee: Junhong Xu
When the RS crashed, we replay WALs to generate recover edits or HFile
directly. If the replaying WAL RS crashed again, the file just writing to may
be corrupted. In some cases, we may want to move on(e.g. in the case of sink to
hfile as we have WAL and replaying the WAL again is OK), and move the file with
extra timestamp as suffix.But if the region is opened again, the corrupted file
can't be opened, and renamed with an extra timestamp again.After some round
like this, the file name will be too long to rename.The log is like this:
{code:java}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$PathComponentTooLongException):
The maximum path component name limit of 6537855
8b0444c27a9d21fb0f4e4293f.1602831270772.1602831291050.1602831296855.1602831408803.1602831493989.1602831584077.1602831600838.1602831659805.1602831736374.1602831738002.1
602831959867.1602831979707.1602832095288.1602832103908.1602832538224.1602833079431
in directory /hbase/XXX/data/default/IntegrationTestBigLinkedList/aa376ec
f026a5e63d0703384e34ec6aa/meta/recovered.hfiles is exceeded: limit=255
length=256
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength(FSDirectory.java:1230)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.verifyFsLimitsForRename(FSDirRenameOp.java:98)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.unprotectedRenameTo(FSDirRenameOp.java:191)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameTo(FSDirRenameOp.java:493)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameToInt(FSDirRenameOp.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3080)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:1113)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:665)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:916)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:862)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2742) at
org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy17.rename(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.rename(ClientNamenodeProtocolTranslatorPB.java:504)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
at com.sun.proxy.$Proxy18.rename(Unknown Source)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy21.rename(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1996)
at
org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:605)
at
org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:226)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
