[
https://issues.apache.org/jira/browse/HDFS-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
abhishek sahani updated HDFS-14947:
-----------------------------------
Priority: Critical (was: Blocker)
> infrequent data loss due to rename functionality breaking
> ---------------------------------------------------------
>
> Key: HDFS-14947
> URL: https://issues.apache.org/jira/browse/HDFS-14947
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.7.3
> Reporter: abhishek sahani
> Priority: Critical
>
> We are facing an issue where data is getting lost from hdfs during rename ,
> in namenode logs we check file is renamed successfully but in hdfs after
> rename file is not present at destination location and thus we are loosing
> the data.
>
> namenode logs:
> 19/10/31 16:54:09 DEBUG top.TopAuditLogger: ------------------- logged event
> for top service: allowed=true ugi=root (auth:SIMPLE) ip=/*.*.*.* cmd=rename
> src=/topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
>
> dst=/topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
> perm=root:supergroup:rw-r--r--
>
> 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* NameSystem.renameTo:
> /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
> to
> /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
> 19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 8 on 9000: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> *.*.*.*:39854 Call#48333 Retry#0
> 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* FSDirectory.renameTo:
> /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
> to
> /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
> 19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 6 on 9000:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from *.*.*.*:39854
> Call#48337 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
> 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo:
> /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet
> is renamed to
> /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
