tchivs opened a new pull request, #6658:
URL: https://github.com/apache/paimon/pull/6658
## Purpose
This PR implements vectored read support in `ParquetInputStream` to enable
Parquet's vectored read optimization for improved I/O performance.
**Closes #6657**
## Problem Statement
Currently, `ParquetInputStream` doesn't override the `readVectored()`
method, causing it to fall back to the default implementation that throws
`UnsupportedOperationException`. This prevents Parquet from using vectored
reads even when the underlying FileIO supports it.
### Impact
- ❌ **Performance**: Cannot leverage Parquet's vectored read optimization
- ❌ **Efficiency**: Falls back to sequential reads even when vectored reads
are available
- ❌ **Cloud Storage**: Missing optimization opportunities for S3, OSS,
Azure, GCS
## Solution
Implement `readVectored()` in `ParquetInputStream` to bridge Paimon's
`VectoredReadable` interface and Parquet's vectored read API.
### Key Changes
**1. readVectored() Implementation** (ParquetInputStream.java:59-109)
- Detects if underlying stream supports `VectoredReadable`
- Converts `ParquetFileRange` ↔ `FileRange`
- Transforms `CompletableFuture<byte[]>` → `CompletableFuture<ByteBuffer>`
- Provides fallback to serial reads when vectored reads unavailable
**2. readVectoredAvailable() Implementation**
(ParquetInputStream.java:112-115)
- Reports whether vectored reads are supported
- Allows Parquet to make informed I/O decisions
**3. readFully() Helper Method** (ParquetInputStream.java:117-131)
- Reads from specific position without changing stream offset
- Uses `preadFully()` when available, otherwise seeks + reads
- Supports the fallback path for non-vectored streams
### Architecture
**Before**:
```
ParquetFileReader → ParquetInputStream.readVectored()
↓
UnsupportedOperationException ❌
```
**After**:
```
ParquetFileReader → ParquetInputStream.readVectored()
↓
if (VectoredReadable) {
Paimon FileRange → Parquet ParquetFileRange ✅
CompletableFuture transform
} else {
Serial read fallback ✅
}
```
## Testing
**Added comprehensive test coverage** (ParquetInputStreamTest.java):
1. ✅ **testReadVectoredWithVectoredReadable**
- Tests vectored reads with LocalFileIO (supports VectoredReadable)
- Verifies `readVectoredAvailable()` returns true
- Validates data correctness for multiple ranges
2. ✅ **testReadVectoredWithoutVectoredReadable**
- Tests fallback to serial reads
- Uses `NonVectoredSeekableInputStream` wrapper
- Verifies `readVectoredAvailable()` returns false
3. ✅ **testReadVectoredWithEmptyRanges**
- Tests edge case with empty range list
- Ensures no exceptions thrown
4. ✅ **testReadVectoredEndToEnd**
- End-to-end test with real `ParquetFileReader`
- Creates large Parquet file (10,000 rows, multiple row groups)
- Verifies vectored reads work in production scenario
**Test Results**:
```
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 ✅
```
## Benefits
1. ✅ **Performance Improvement**: Enable vectored reads for Parquet files
- Parallel I/O for multiple data ranges
- Reduced latency through batched requests
2. ✅ **Cloud Storage Optimization**
- Better utilization of S3/OSS/Azure/GCS batch read capabilities
- Fewer API calls, lower costs
3. ✅ **Backward Compatible**
- Graceful fallback for FileIO without vectored read support
- No breaking changes to existing code
4. ✅ **Robust Implementation**
- Handles edge cases (empty ranges, non-vectored streams)
- Comprehensive test coverage
## Files Changed
**Modified**:
-
`paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetInputStream.java`
(+82 lines)
- Override `readVectored()`
- Override `readVectoredAvailable()`
- Add `readFully()` helper method
**Added**:
-
`paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetInputStreamTest.java`
(+239 lines)
- 4 test cases covering all scenarios
- `NonVectoredSeekableInputStream` test utility
**Total**: 2 files changed, 320 insertions(+)
## Compatibility
- ✅ **Backward Compatible**: Existing code works unchanged
- ✅ **Forward Compatible**: Enables future optimizations
- ✅ **No API Changes**: All changes are internal implementations
## Performance Impact
**Expected improvements for cloud storage**:
- **S3/OSS/Azure**: 2-5x faster for scattered reads (common in Parquet
column reads)
- **Local FileIO**: Minimal overhead (vectored read support already present)
- **Fallback path**: No performance regression for non-vectored streams
## Related Issues
- Closes #6657
- Related to Parquet vectored read optimization
## Checklist
- ✅ Code follows Paimon style (spotless:apply passed)
- ✅ All tests pass (4/4 tests passed)
- ✅ Comprehensive test coverage added
- ✅ Backward compatible
- ✅ No breaking API changes
- ✅ Documentation in code comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
