[
https://issues.apache.org/jira/browse/HADOOP-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-20059:
------------------------------------
Labels: pull-request-available (was: )
> AbstractFileSystem#create skips the blockSize/bytesPerChecksum check unless
> the BytesPerChecksum option is set
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20059
> URL: https://issues.apache.org/jira/browse/HADOOP-20059
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Yang Jie
> Priority: Minor
> Labels: pull-request-available
>
> AbstractFileSystem#create rejects a block size that is not a multiple of the
> checksum size with {{if (blockSize % bytesPerChecksum != 0)}}.
> {{bytesPerChecksum}} is a local variable that stays at -1 unless the caller
> passes {{CreateOpts.bytesPerChecksum}}, and {{x % -1}} is always 0, so the
> check never fires in the usual case. HADOOP-8240 removed the line that used
> to fill in the server default and left the check reading the old variable.
> On HDFS, {{FileContext.create(p, EnumSet.of(CREATE, OVERWRITE),
> CreateOpts.blockSize(1000))}} gets past AbstractFileSystem and the NameNode
> creates the file, deleting the existing one because of OVERWRITE. Only then
> does the DFSOutputStream constructor throw "bytesPerChecksum ... must divide
> block size". The caller gets an exception, the old file is gone, and an empty
> file stays under construction until lease recovery.
> The fix checks against {{checksumOpt.getBytesPerChecksum()}}, the value the
> stream is created with, so the create fails before it reaches the file system.
> On file systems that ignore the block size (local, object stores),
> FileContext#create with a block size that is not a multiple of the checksum
> size (512 by default) now fails, as it did before HADOOP-8240.
> DistributedFileSystem#create has the same create-then-check order inside
> DFSClient; that needs a separate HDFS change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
