abhishek sahani created HDFS-14947:
--------------------------------------
Summary: infrequent data loss due to rename functionality breaking
Key: HDFS-14947
URL: https://issues.apache.org/jira/browse/HDFS-14947
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.7.3
Reporter: abhishek sahani
We are facing an issue where data is getting lost from hdfs during rename , in
namenode logs we check file is renamed successfully but in hdfs after rename
file is not present at destination location and thus we are loosing the data.
namenode logs:
19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* NameSystem.renameTo:
/topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
to
/topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 8 on 9000: responding to
org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from *.*.*.*:39854
Call#48333 Retry#0
19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* FSDirectory.renameTo:
/topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
to
/topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 6 on 9000:
org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from *.*.*.*:39854
Call#48337 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* FSDirectory.unprotectedRenameTo:
/topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
is renamed to
/topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
