[
https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913832#comment-16913832
]
Jungtaek Lim commented on HADOOP-16255:
---------------------------------------
[[email protected]]
Spark also hits this issue https://issues.apache.org/jira/browse/SPARK-28025
though the concern is a bit different. Once "crc for src" is not renamed, it's
a leak as we have no opaque way to delete this. We'll just delete the crc file
as an workaround but definitely looking forward to remove the hack.
Btw, the Hadoop version Spark is using is 2.7.4, but it seems to be a long
standing issue through various version lines (maybe all 2.x versions, or even
earlier?). What would be the preferred way to do this? Filing new issue
describing Spark's case, or could you please add 2.7.4 to the affected
version/s here?
> ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
> ------------------------------------------------------------------------------
>
> Key: HADOOP-16255
> URL: https://issues.apache.org/jira/browse/HADOOP-16255
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 3.1.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum
> with the file.
> As a result, if a file is renamed over an existing file using rename(src,
> dest, OVERWRITE) the renamed file will be considered to have an invalid
> checksum -the old one is picked up instead.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
