[
https://issues.apache.org/jira/browse/HADOOP-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778545#comment-16778545
]
Eric Yang commented on HADOOP-16107:
------------------------------------
Append seems like an unsupported operation for ChecksumFileSystem, and same
applies to concat. It is not clear if someone actually used those API.
ChecksumFileSystem has the option to skip checksum since HADOOP-8042. It is
likely that ChecksumFileSystem can be called without actually doing any crc
check, which can defeat HDFS's purpose to be a checksum file system. Let's
hope the use cases that skipped crc know what they were doing without
corrupting data. I can commit the patch if white spaces are fixed.
> FilterFileSystem doesn't wrap all create() or new builder calls; may skip CRC
> logic
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-16107
> URL: https://issues.apache.org/jira/browse/HADOOP-16107
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 3.0.3, 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-16107-001.patch, HADOOP-16107-003.patch
>
>
> LocalFS is a subclass of filterFS, but overrides create and open so that
> checksums are created and read.
> MAPREDUCE-7184 has thrown up that the new builder openFile() call is being
> forwarded to the innerFS without CRC checking. Reviewing/fixing that has
> shown that some of the create methods aren't being correctly wrapped, so not
> generating CRCs
> * createFile() builder
> The following create calls
> {code}
> public FSDataOutputStream createNonRecursive(final Path f,
> final FsPermission permission,
> final EnumSet<CreateFlag> flags,
> final int bufferSize,
> final short replication,
> final long blockSize,
> final Progressable progress) throws IOException;
> public FSDataOutputStream create(final Path f,
> final FsPermission permission,
> final EnumSet<CreateFlag> flags,
> final int bufferSize,
> final short replication,
> final long blockSize,
> final Progressable progress,
> final Options.ChecksumOpt checksumOpt) throws IOException {
> return super.create(f, permission, flags, bufferSize, replication,
> blockSize, progress, checksumOpt);
> }
> {code}
> This means that applications using these methods, directly or indirectly to
> create files aren't actually generating checksums.
> Fix: implement these methods & relay to local create calls, not to the inner
> FS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
