[
https://issues.apache.org/jira/browse/KUDU-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783824#comment-17783824
]
Alexey Serbin edited comment on KUDU-3523 at 11/8/23 1:11 AM:
--------------------------------------------------------------
[~wangxixu],
Thank you for reporting the issue! Indeed, as per
[1|https://www.man7.org/linux/man-pages/man2/stat.2.html], the output of the
{{stat}} utility reports on different 'block sizes' when run on the file and on
the filesystem level (that's why there is {{\-f}} option).
That's right: st_blksize isn't not supposed to be always equal to the
filesystem block size, and that's so "by design", AFAIK. The 'st_blksize'
stands for the IO block size, or "preferred" block size for efficient file
system IO (a.k.a. optimal IO transfer size hint), see
[2|https://www.man7.org/linux/man-pages/man2/statx.2.html].
As per filesystem operations and {{LogBlockManager}} in Kudu, one crucial
invariant is that 'st_blksize' is a multiple of the filesystem's block size.
At least, such invariant is important in the scope of addressing
[KUDU-620|https://issues.apache.org/jira/browse/KUDU-620] (that's where the
{{PathInstanceMetadataPB::filesystem_block_size_bytes}} field has originated
from).
In the scope of this JIRA, please feel free to add references to particular
places where the difference between the IO block and the filesystem block sizes
might lead to inconsistencies. I guess one of those is the misleading name of
the {{PathInstanceMetadataPB::filesystem_block_size_bytes}} field. Probably,
there are more places where the difference is important, and that could lead to
issues in the actual functionality.
#
[https://www.man7.org/linux/man-pages/man2/stat.2.html|https://www.man7.org/linux/man-pages/man2/stat.2.html]
#
[https://www.man7.org/linux/man-pages/man2/statx.2.html|https://www.man7.org/linux/man-pages/man2/statx.2.html]
was (Author: aserbin):
[~wangxixu],
Thank you for reporint the issue! Indeed, as per
[1|https://www.man7.org/linux/man-pages/man2/stat.2.html], the output of the
{{stat}} utility reports on different 'block sizes' when run on the file and on
the filesystem level (that's why there is {{\-f}} option).
That's right: st_blksize isn't not supposed to be always equal to the
filesystem block size, and that's so "by design", AFAIK. The 'st_blksize'
stands for the IO block size, or "preferred" block size for efficient file
system IO (a.k.a. optimal IO transfer size hint), see
[2|https://www.man7.org/linux/man-pages/man2/statx.2.html].
As per filesystem operations and {{LogBlockManager}} in Kudu, one crucial
invariant is that 'st_blksize' is a multiple of the filesystem's block size.
At least, such invariant is important in the scope of addressing
[KUDU-620|https://issues.apache.org/jira/browse/KUDU-620] (that's where the
{{PathInstanceMetadataPB::filesystem_block_size_bytes}} field has originated
from).
In the scope of this JIRA, please feel free to add references to particular
places where the difference between the IO block and the filesystem block sizes
might lead to inconsistencies. I guess one of those is the misleading name of
the {{PathInstanceMetadataPB::filesystem_block_size_bytes}} field. Probably,
there are more places where the difference is important, and that could lead to
issues in the actual functionality.
#
[https://www.man7.org/linux/man-pages/man2/stat.2.html|https://www.man7.org/linux/man-pages/man2/stat.2.html]
#
[https://www.man7.org/linux/man-pages/man2/statx.2.html|https://www.man7.org/linux/man-pages/man2/statx.2.html]
> st_blksize is not alway equal to the filesystem block size
> ----------------------------------------------------------
>
> Key: KUDU-3523
> URL: https://issues.apache.org/jira/browse/KUDU-3523
> Project: Kudu
> Issue Type: Bug
> Reporter: Xixu Wang
> Priority: Major
> Attachments: image-2023-11-06-15-42-46-082.png,
> image-2023-11-06-15-45-11-819.png, image-2023-11-06-15-45-39-233.png,
> image-2023-11-06-15-52-41-834.png
>
>
> In my ** aarch64 architecture system, the st_blksize is not equal to the real
> filesystem block size. The st_blksize in my system is 65536 bytes, but the
> block size of the filesystem is 4096 bytes. When writing some data which size
> is less than 4096 bytes, the file on disk size is 4096 bytes not 65536 bytes.
> But in kudu, it use st_blksize to decide the filesystem block size, which is
> not always right.
>
> *1. The test environment*
> Linux hybrid01 4.19.90-23.30.v2101.ky10.aarch64 #1 SMP Thu Dec 15 09:57:55
> CST 2022 aarch64 aarch64 aarch64 GNU/Linux. And a docker container runs on it.
> *2.Create a file with encryption header*
>
> {code:java}
> const string kFile = JoinPathSegments(test_dir_, "encrypted_file");
> unique_ptr<RWFile> rw;
> RWFileOptions opts;
> opts.is_sensitive = true;
> ASSERT_OK(env_->NewRWFile(opts, kFile, &rw));
> uint64_t file_size = 0;
> env_->GetFileSizeOnDisk(kFile, &file_size); {code}
> *3.stat the file*
>
> The IO Block size is 65536, which means st_blsize is 65536, the file logic
> size is 64 bytes.
> !image-2023-11-06-15-42-46-082.png!
> *4. filesystem block size is 4096 bytes*
> !image-2023-11-06-15-45-39-233.png!
> *5.The file on disk size is 4096 bytes*
> !image-2023-11-06-15-52-41-834.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
