[
https://issues.apache.org/jira/browse/HADOOP-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138454#comment-18138454
]
ASF GitHub Bot commented on HADOOP-20059:
-----------------------------------------
LuciferYang opened a new pull request, #8922:
URL: https://github.com/apache/hadoop/pull/8922
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-20059
`AbstractFileSystem#create` rejects a block size that is not a multiple of
the checksum size by checking `blockSize % bytesPerChecksum`.
`bytesPerChecksum` is a local that stays at -1 unless the caller passes
`CreateOpts.bytesPerChecksum`, and `x % -1` is always 0, so the check never
fires in the usual case. On HDFS, `FileContext#create` with such a block size
reaches the NameNode, which creates the file (or deletes the old one with
`OVERWRITE`), and only then does `DFSOutputStream` throw.
This patch checks against `checksumOpt.getBytesPerChecksum()`, the value the
stream is created with. On file systems that ignore the block size,
`FileContext#create` with a block size that is not a multiple of the checksum
size (512 by default) now fails, as it did before HADOOP-8240.
`DistributedFileSystem#create` has the same create-then-check order inside
`DFSClient`, which needs a separate HDFS change.
### How was this patch tested?
Added
`FileContextCreateMkdirBaseTest#testCreateBlockSizeNotMultipleOfChecksumSize`:
creating with a block size of 1000 throws `HadoopIllegalArgumentException` and
leaves no file behind. Without the fix the create succeeds on the local and
viewfs subclasses. The case is in the shared base class, so it also runs in the
S3A, OSS and ADL FileContext tests, which I did not run against a store.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
- [x] The PR includes the phrase "Contains content generated by Claude Opus".
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Opus.
> AbstractFileSystem#create skips the blockSize/bytesPerChecksum check unless
> the BytesPerChecksum option is set
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20059
> URL: https://issues.apache.org/jira/browse/HADOOP-20059
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Yang Jie
> Priority: Minor
>
> AbstractFileSystem#create rejects a block size that is not a multiple of the
> checksum size with {{if (blockSize % bytesPerChecksum != 0)}}.
> {{bytesPerChecksum}} is a local variable that stays at -1 unless the caller
> passes {{CreateOpts.bytesPerChecksum}}, and {{x % -1}} is always 0, so the
> check never fires in the usual case. HADOOP-8240 removed the line that used
> to fill in the server default and left the check reading the old variable.
> On HDFS, {{FileContext.create(p, EnumSet.of(CREATE, OVERWRITE),
> CreateOpts.blockSize(1000))}} gets past AbstractFileSystem and the NameNode
> creates the file, deleting the existing one because of OVERWRITE. Only then
> does the DFSOutputStream constructor throw "bytesPerChecksum ... must divide
> block size". The caller gets an exception, the old file is gone, and an empty
> file stays under construction until lease recovery.
> The fix checks against {{checksumOpt.getBytesPerChecksum()}}, the value the
> stream is created with, so the create fails before it reaches the file system.
> On file systems that ignore the block size (local, object stores),
> FileContext#create with a block size that is not a multiple of the checksum
> size (512 by default) now fails, as it did before HADOOP-8240.
> DistributedFileSystem#create has the same create-then-check order inside
> DFSClient; that needs a separate HDFS change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
