asif-moh opened a new issue, #3600:
URL: https://github.com/apache/parquet-java/issues/3600
### Describe the enhancement requested
**Describe the enhancement requested**
`CorruptStatistics.shouldIgnoreStatistics(String createdBy,
PrimitiveTypeName columnType)` performs `VersionParser.parse() `and
`SemanticVersion.parse()` on every invocation. The `createdBy` string is
constant per file (from `FileMetaData.created_by`), but this method is called
once per column chunk per row group during
`ParquetMetadataConverter.fromParquetMetadata()`.
For a file with R row groups and C columns, the version string is parsed R×C
times during file open — all yielding the same result.
**Profiling data**
`shouldIgnoreStatistics` accounts for ~65% of `fromParquetMetadata` CPU time
across multiple samples.
**Impact**
High CPU during `ParquetFileReader` construction on files with many row
groups/columns.
**Proposed fix**
Compute `shouldIgnoreStatistics` once per file before the row group loop in
`fromParquetMetadata`, and pass the pre-computed boolean through
`buildColumnChunkMetaData` → `fromParquetStatisticsInternal`.
Since `buildColumnChunkMetaData` and `fromParquetStatisticsInternal` are
public/package-level API methods enforced by japicmp-maven-plugin, we will add
overloaded methods that accept a boolean `shouldIgnoreBinaryStats` parameter
rather than changing existing signatures. The existing String
`createdBy` signatures remain for backward compatibility and delegate to
the new overloads.
The page-level path in `ParquetFileReader.Chunk.readAllPages()` also calls
`shouldIgnoreStatistics` via `fromParquetStatisticsInternal`, but is not in
scope for this fix — at read time the cost is masked by I/O, decompression, and
decoding. The footer path during file open is where R×C calls happen in a tight
loop with no I/O to amortize the cost.
Affected code path:
- `ParquetMetadataConverter.fromParquetMetadata()` — footer reading (hot
path during open)
### Component(s)
Core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
