BsoBird opened a new pull request, #2718:
URL: https://github.com/apache/orc/pull/2718
### What changes were proposed in this pull request?
- Refactor `RecordReaderUtils.readDiskRangesVectored` to coalesce
`BufferChunk`s through a new pure `coalesceRanges` helper that merges
contiguous/overlapping ranges and, when `minSeekSize > 0`, also folds in small
gaps (`nextOffset - currentEnd <= minSeekSize`), matching the sync path's
`findSingleRead` semantics so vectored I/O no longer regresses on
minSeekSize-bounded workloads
- Enforce a hard per-range cap of `IOUtils.MAX_ARRAY_SIZE` and compute
merged spans in long arithmetic, flushing the in-flight `FileRange` when it
would exceed the cap so offsets near `Integer.MAX_VALUE` no longer wrap to
negative/truncated int lengths
- Treat already-filled chunks (`hasData() == true`) as merge barriers so the
vectored path never re-reads data that is already in memory, and backfill
original chunks from merged range buffers with a single O(chunks + ranges)
forward pass
- Short-circuit single-range batches to a synchronous positioned `readFully`
instead of submitting one `FileRange` to `readVectored`, avoiding executor task
+ `CompletableFuture` overhead where no parallelism is possible
- Reject a negative `minSeekSize` up front (`IllegalArgumentException`)
instead of failing later with a misleading error
- Add 14 unit tests covering gap-0 merge, gap <= / > `minSeekSize`,
`hasData` boundaries, the size cap, negative input rejection, the single-range
short-circuit, and `minSeekSize=0` equivalence; rework `testHadoopVectoredIO`
to keep exercising `readVectored` (a full scan now coalesces to one range)
The non-vectored (sync) read path is unchanged. With default configuration
(`minSeekSize=0`) exactly the same bytes are read as before.
### Why are the changes needed?
When vectored I/O is active (Hadoop >= 3.3.5, zero-copy disabled), the
vectored read path builds one `FileRange` per `BufferChunk` with no merging:
1. **`orc.min.disk.seek.size` is silently ignored.** The sync path merges
ranges whose gap is within `minSeekSize` (`findSingleRead`); the vectored path
does not. `TestMinSeekSize.readAlternateColumnsWMinSeekSize` fails on Hadoop
3.4.1 (reads 5.26% of the file instead of ~106%); it passes on Hadoop 3.3.4,
where vectored I/O is disabled — confirming the vectored path as the cause.
2. **Measurable read-path regression vs the sync path (and ORC 1.5.3) on
files with many small stripes.** Per-range async dispatch overhead dominates
when ranges are tiny and numerous.
Local benchmark (2M rows, 1954 stripes of ~1k rows, ZLIB/NONE, warm page
cache, Hadoop 3.4.1, median of 15 iterations, before → after):
| Mode | ZLIB before | ZLIB after | NONE before | NONE after |
|---------------------|-------------|------------|-------------|------------|
| full scan | 216.9 ms | 187.8 ms | 97.3 ms | 73.3 ms |
| projection | 129.7 ms | 105.4 ms | 51.7 ms | 32.9 ms |
| filter (SARG) | 51.3 ms | 28.2 ms | 43.9 ms | 21.5 ms |
| projection + filter | 34.1 ms | 16.8 ms | 32.8 ms | 14.9 ms |
Read bytes and row counts are identical in every cell; after the fix the
vectored path matches the non-vectored path (and 1.5.3) on this workload.
### How was this patch tested?
- `TestMinSeekSize` now passes with Hadoop 3.4.1 (previously failing; see
above for reproduction via `-Dhadoop.version=3.3.4` vs `3.4.1`).
- 14 new tests in `TestRecordReaderUtils` for `coalesceRanges` (positive +
negative cases listed above) and a new
`TestRecordReaderImpl.testHadoopVectoredIOSingleRangeUsesSyncRead` asserting
single-range batches never call `readVectored`.
- Full `java/core` suite on main: **1558 tests, 0 failures, 0 errors** (21
skipped): `cd java && ./mvnw -pl core test`
- Benchmark reproduction: read a 1954-stripe ORC file (ZLIB and NONE) 15
times per mode (scan / projection / SARG filter / projection+filter) with
Hadoop 3.4.1 on the classpath; verify identical `read_bytes` and `rows_read`
between vectored and non-vectored paths.
### Was this patch authored or co-authored using generative AI tooling?
co-authored-by: Kimi K3 via OpenCode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
