weijietong opened a new pull request, #10836:
URL: https://github.com/apache/paimon/pull/10836
### Purpose
This PR introduces a customized Parquet footer parser to significantly
improve metadata reading performance, especially for wide tables with hundreds
to thousands of columns.
#### Background
When reading Parquet files, Paimon needs to parse the footer metadata
serialized with Thrift Compact Protocol. For wide tables (hundreds to thousands
of columns), footer parsing becomes a significant performance bottleneck. The
official ParquetMetadataConverter materializes every column chunk's metadata
including all Statistics, even when only a subset of
columns is needed.
#### Implementation
The selective footer parser walks the Thrift compact-protocol bytes
directly with a hand-written ThriftCompactReader and:
- Selective materialization: Only materializes column chunks whose
top-level column is requested. Other column chunks are still scanned (the
compact protocol forces a full byte pass) but their bytes are not retained.
- Statistics skipping: Optionally skips Statistics parsing for kept
columns (AI training reads never consume footer statistics since stats-based
row-group filtering requires a pushdown filter).
- Schema preservation: Always materializes the complete schema (including
full LogicalType union) regardless of the column subset.
- Fallback mechanism: Automatically falls back to the official parser when
encountering encrypted footers, encrypted columns, or unexpected structures.
Object construction is delegated to
ParquetMetadataConverter.fromParquetMetadata() to reuse the battle-tested
conversion, ensuring byte-for-byte equivalence with the official reader for the
retained subset.
#### Performance Results
Benchmark on a 232MB file with 1,570 columns and 8 row groups (1.29MB
footer):
| Requested Columns | Official Parser | Selective Parser | Speedup |
Memory Saved |
| ----------------- | --------------- | ---------------- | ------- |
------------ |
| 10 (0.6%) | 21.11 ms | 2.46 ms | 8.6x | 54%
|
| 50 (3.2%) | 21.65 ms | 2.62 ms | 8.3x | 53%
|
| 100 (6.4%) | 21.06 ms | 2.90 ms | 7.3x | 52%
|
| 500 (31.8%) | 21.58 ms | 4.50 ms | 4.8x | 42%
|
| 1000 (63.7%) | 20.62 ms | 6.26 ms | 3.3x | 29%
|
| 1570 (100%) | 20.21 ms | 8.47 ms | 2.4x | 14%
|
Even in full-parse mode (all columns requested), the selective parser
achieves 2.4x speedup due to:
- Hand-written parser vs Thrift-generated code (avoiding reflection and
setter overhead)
- Direct byte array access vs unbuffered TIOStreamTransport (avoiding
50,000+ byte[] allocations for Statistics)
- No String.intern() calls (avoiding global String pool synchronization)
- Single-phase parsing (avoiding Thrift tree memory)
- Avoiding ~8 wrapper objects per column chunk
#### Integration
The selective parser is enabled in ParquetReaderFactory when no pushdown
predicates are present (since predicates require Statistics for row-group
filtering). It's controlled by two configuration properties:
- paimon.parquet.footer.requested.columns: Comma-separated list of
top-level column names to retain
- paimon.parquet.footer.skip.statistics: Whether to skip Statistics
parsing (default: true for AI training workloads)
### Tests
Added comprehensive test coverage in SelectiveFooterParserTest with 30
test cases:
Equivalence Tests
- Full parse equivalence with official NO_FILTER parse
- Selective subset parse correctness
- Dropped columns do not affect block-level info
- Skip statistics parse mode
Complex Schema Tests
- Deep nesting (3+ levels)
- Map type (key-value structures)
- Array type (list structures)
Boundary Condition Tests
- Empty file (0 rows)
- Single row group
- All columns requested
- Non-existent columns requested
Exception Path Tests
- Malformed footer (truncated)
- Malformed footer (wrong thrift type)
- Fallback signal handling
Encryption Tests
- Encrypted footer fallback verification
- Encrypted column fallback verification
Integration Tests
- ParquetReadOptions integration
- Split requested columns utility
- Many row groups (1000 rows, small row group size)
- Selective parse with nested columns
All 30 tests pass successfully, validating byte-for-byte equivalence with
the official parser for the retained subset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
