[
https://issues.apache.org/jira/browse/ORC-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yongzhi.shao updated ORC-2224:
------------------------------
Description:
Summary:
Vectored read path ignores orc.min.disk.seek.size and pays per-range async
overhead; coalesce ranges and read single-range batches synchronously
Type: Bug
Component: Java (core reader)
Affects Version: 2.1.x, 3.0.0-SNAPSHOT (only when vectored IO is active: Hadoop
>= 3.3.5, zero-copy disabled)
Description:
h3. Problem
When the vectored read path ({{{}RecordReaderUtils.readDiskRangesVectored{}}})
is active:
# {{orc.min.disk.seek.size}} is silently ignored. The sync path merges disk
ranges whose gap is within {{minSeekSize}} ({{{}findSingleRead{}}}), but the
vectored path builds one {{FileRange}} per {{BufferChunk}} without any merging.
As a result, {{TestMinSeekSize.readAlternateColumnsWMinSeekSize}} *fails on
Hadoop >= 3.3.5* (reads 5.26% of the file instead of ~106%) and any
user-configured {{orc.min.disk.seek.size}} has no effect.
# Ranges that are contiguous or overlapping within the same batch are still
submitted as separate \{{FileRange}}s, paying one async task + future per range.
# A batch that consists of a single range is still submitted to
{{{}readVectored{}}}, which adds executor task + {{CompletableFuture}} overhead
with no parallelism benefit — the caller blocks on it immediately.
h3. Observed impact
Local benchmark: 2M rows, 1954 stripes of ~1k rows each, ZLIB and NONE, warm
page cache, Hadoop 3.4.1, median of 15 iterations, ORC 2.1.2 before → after:
||Mode||ZLIB before||ZLIB after||NONE before||NONE after||
|full scan|216.9 ms|187.8 ms|97.3 ms|73.3 ms|
|projection|129.7 ms|105.4 ms|51.7 ms|32.9 ms|
|filter (SARG)|51.3 ms|28.2 ms|43.9 ms|21.5 ms|
|projection + filter|34.1 ms|16.8 ms|32.8 ms|14.9 ms|
PR: [ORC-2224:Coalesce vectored read ranges honoring minSeekSize and read
single-range batches synchronously|https://github.com/apache/orc/pull/2718]
was:
Summary:
Vectored read path ignores orc.min.disk.seek.size and pays per-range async
overhead; coalesce ranges and read single-range batches synchronously
Type: Bug
Component: Java (core reader)
Affects Version: 2.1.x, 3.0.0-SNAPSHOT (only when vectored IO is active: Hadoop
>= 3.3.5, zero-copy disabled)
Description:
h3. Problem
When the vectored read path (\{{RecordReaderUtils.readDiskRangesVectored}}) is
active:
# \{{orc.min.disk.seek.size}} is silently ignored. The sync path merges disk
ranges whose gap is within \{{minSeekSize}} (\{{findSingleRead}}), but the
vectored path builds one \{{FileRange}} per \{{BufferChunk}} without any
merging. As a result, \{{TestMinSeekSize.readAlternateColumnsWMinSeekSize}}
*fails on Hadoop >= 3.3.5* (reads 5.26% of the file instead of ~106%) and any
user-configured \{{orc.min.disk.seek.size}} has no effect.
# Ranges that are contiguous or overlapping within the same batch are still
submitted as separate \{{FileRange}}s, paying one async task + future per range.
# A batch that consists of a single range is still submitted to
\{{readVectored}}, which adds executor task + \{{CompletableFuture}} overhead
with no parallelism benefit — the caller blocks on it immediately.
h3. Observed impact
Local benchmark: 2M rows, 1954 stripes of ~1k rows each, ZLIB and NONE, warm
page cache, Hadoop 3.4.1, median of 15 iterations, ORC 2.1.2 before → after:
||Mode||ZLIB before||ZLIB after||NONE before||NONE after|
|full scan|216.9 ms|187.8 ms|97.3 ms|73.3 ms|
|projection|129.7 ms|105.4 ms|51.7 ms|32.9 ms|
|filter (SARG)|51.3 ms|28.2 ms|43.9 ms|21.5 ms|
|projection + filter|34.1 ms|16.8 ms|32.8 ms|14.9 ms
> [Java] Vectored read coalescing honors minSeekSize; single-range batches read
> synchronously
> -------------------------------------------------------------------------------------------
>
> Key: ORC-2224
> URL: https://issues.apache.org/jira/browse/ORC-2224
> Project: ORC
> Issue Type: Improvement
> Reporter: yongzhi.shao
> Priority: Major
>
> Summary:
> Vectored read path ignores orc.min.disk.seek.size and pays per-range async
> overhead; coalesce ranges and read single-range batches synchronously
> Type: Bug
> Component: Java (core reader)
> Affects Version: 2.1.x, 3.0.0-SNAPSHOT (only when vectored IO is active:
> Hadoop >= 3.3.5, zero-copy disabled)
> Description:
> h3. Problem
> When the vectored read path
> ({{{}RecordReaderUtils.readDiskRangesVectored{}}}) is active:
> # {{orc.min.disk.seek.size}} is silently ignored. The sync path merges disk
> ranges whose gap is within {{minSeekSize}} ({{{}findSingleRead{}}}), but the
> vectored path builds one {{FileRange}} per {{BufferChunk}} without any
> merging. As a result, {{TestMinSeekSize.readAlternateColumnsWMinSeekSize}}
> *fails on Hadoop >= 3.3.5* (reads 5.26% of the file instead of ~106%) and any
> user-configured {{orc.min.disk.seek.size}} has no effect.
> # Ranges that are contiguous or overlapping within the same batch are still
> submitted as separate \{{FileRange}}s, paying one async task + future per
> range.
> # A batch that consists of a single range is still submitted to
> {{{}readVectored{}}}, which adds executor task + {{CompletableFuture}}
> overhead with no parallelism benefit — the caller blocks on it immediately.
> h3. Observed impact
> Local benchmark: 2M rows, 1954 stripes of ~1k rows each, ZLIB and NONE, warm
> page cache, Hadoop 3.4.1, median of 15 iterations, ORC 2.1.2 before → after:
> ||Mode||ZLIB before||ZLIB after||NONE before||NONE after||
> |full scan|216.9 ms|187.8 ms|97.3 ms|73.3 ms|
> |projection|129.7 ms|105.4 ms|51.7 ms|32.9 ms|
> |filter (SARG)|51.3 ms|28.2 ms|43.9 ms|21.5 ms|
> |projection + filter|34.1 ms|16.8 ms|32.8 ms|14.9 ms|
> PR: [ORC-2224:Coalesce vectored read ranges honoring minSeekSize and read
> single-range batches synchronously|https://github.com/apache/orc/pull/2718]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
