[
https://issues.apache.org/jira/browse/HADOOP-19291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887611#comment-17887611
]
Mukund Thakur edited comment on HADOOP-19291 at 10/8/24 2:37 PM:
-----------------------------------------------------------------
ChecksumFileSystem creates the chunked ranges based on the checksum chunk size
and then calls
readVectored on Raw Local which may lead to overlapping ranges even if initial
ranges are disjoint in some cases.
Example :
Range1: 30918-14251143
Range2: 14251143-2958570
If checksum size is 100, checksum ranges become
30900 - 14251200
14251100 - 2958600
which are overlapping.
For more details see comments in [https://github.com/apache/hadoop/pull/7079]
was (Author: mthakur):
ChecksumFileSystem creates the chunked ranges based on the checksum chunk size
and then calls
readVectored on Raw Local which may lead to overlapping ranges even if initial
ranges are disjoint in some cases.
For more details see comments in [https://github.com/apache/hadoop/pull/7079]
> `CombinedFileRange.merge` should not convert disjoint ranges into overlapped
> ones
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-19291
> URL: https://issues.apache.org/jira/browse/HADOOP-19291
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.3.9, 3.5.0, 3.4.1
> Reporter: Dongjoon Hyun
> Priority: Major
> Attachments: Screenshot 2024-09-28 at 22.02.01.png
>
>
> Currently, Hadoop has a bug to convert disjoint ranges into overlapped ones
> and eventually fails by itself.
> {code:java}
> + public void testMergeSortedRanges() {
> + List<FileRange> input = asList(
> + createFileRange(13816220, 24, null),
> + createFileRange(13816244, 7423960, null)
> + );
> + assertIsNotOrderedDisjoint(input, 100, 800);
> + final List<CombinedFileRange> outputList = mergeSortedRanges(
> + sortRangeList(input), 100, 1001, 2500);
> +
> + assertRangeListSize(outputList, 1);
> + assertFileRange(outputList.get(0), 13816200, 7424100);
> + }
> {code}
> !Screenshot 2024-09-28 at 22.02.01.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
