asifsmohammed opened a new issue, #3601: URL: https://github.com/apache/parquet-java/issues/3601
### Describe the enhancement requested `CorruptStatistics.shouldIgnoreStatistics(String createdBy, PrimitiveTypeName columnType)` performs `VersionParser.parse() `and `SemanticVersion.parse()` on every invocation. The `createdBy` string is constant per file (from `FileMetaData.created_by`), but this method is called once per column chunk per row group during `ParquetMetadataConverter.fromParquetMetadata()`. For a file with R row groups and C columns, the version string is parsed R×C times during file open — all yielding the same result. **Profiling data** `shouldIgnoreStatistics` accounts for ~65% of `fromParquetMetadata` CPU time across multiple samples. **Impact** High CPU during `ParquetFileReader` construction on files with many row groups/columns. **Proposed fix** Compute `shouldIgnoreStatistics` once per file before the row group loop in `fromParquetMetadata`, and pass the pre-computed boolean through `buildColumnChunkMetaData` → `fromParquetStatisticsInternal`. Since `buildColumnChunkMetaData` and `fromParquetStatisticsInternal` are public/package-level API methods enforced by japicmp-maven-plugin, we will add overloaded methods that accept a boolean `shouldIgnoreBinaryStats` parameter rather than changing existing signatures. The existing String `createdBy` signatures remain for backward compatibility and delegate to the new overloads. The page-level path in `ParquetFileReader.Chunk.readAllPages()` also calls `shouldIgnoreStatistics` via `fromParquetStatisticsInternal`, but is not in scope for this fix — at read time the cost is masked by I/O, decompression, and decoding. The footer path during file open is where R×C calls happen in a tight loop with no I/O to amortize the cost. Affected code path: - `ParquetMetadataConverter.fromParquetMetadata()` — footer reading (hot path during open) ### Component(s) Core -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
