yongzhi.shao created ORC-2224:
---------------------------------
Summary: [Java] Vectored read coalescing honors minSeekSize;
single-range batches read synchronously
Key: ORC-2224
URL: https://issues.apache.org/jira/browse/ORC-2224
Project: ORC
Issue Type: Improvement
Reporter: yongzhi.shao
Summary:
Vectored read path ignores orc.min.disk.seek.size and pays per-range async
overhead; coalesce ranges and read single-range batches synchronously
Type: Bug
Component: Java (core reader)
Affects Version: 2.1.x, 3.0.0-SNAPSHOT (only when vectored IO is active: Hadoop
>= 3.3.5, zero-copy disabled)
Description:
h3. Problem
When the vectored read path (\{{RecordReaderUtils.readDiskRangesVectored}}) is
active:
# \{{orc.min.disk.seek.size}} is silently ignored. The sync path merges disk
ranges whose gap is within \{{minSeekSize}} (\{{findSingleRead}}), but the
vectored path builds one \{{FileRange}} per \{{BufferChunk}} without any
merging. As a result, \{{TestMinSeekSize.readAlternateColumnsWMinSeekSize}}
*fails on Hadoop >= 3.3.5* (reads 5.26% of the file instead of ~106%) and any
user-configured \{{orc.min.disk.seek.size}} has no effect.
# Ranges that are contiguous or overlapping within the same batch are still
submitted as separate \{{FileRange}}s, paying one async task + future per range.
# A batch that consists of a single range is still submitted to
\{{readVectored}}, which adds executor task + \{{CompletableFuture}} overhead
with no parallelism benefit — the caller blocks on it immediately.
h3. Observed impact
Local benchmark: 2M rows, 1954 stripes of ~1k rows each, ZLIB and NONE, warm
page cache, Hadoop 3.4.1, median of 15 iterations, ORC 2.1.2 before → after:
||Mode||ZLIB before||ZLIB after||NONE before||NONE after|
|full scan|216.9 ms|187.8 ms|97.3 ms|73.3 ms|
|projection|129.7 ms|105.4 ms|51.7 ms|32.9 ms|
|filter (SARG)|51.3 ms|28.2 ms|43.9 ms|21.5 ms|
|projection + filter|34.1 ms|16.8 ms|32.8 ms|14.9 ms
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
