asifsmohammed commented on PR #3607: URL: https://github.com/apache/parquet-java/pull/3607#issuecomment-4651698708
I generally like the approach @yadavay-amzn, here's why I would not use cache alone as I mentioned in the issue. It's better than the current approach of checking it RGxCC times - In high workload scenarios we are going to call `shouldIgnoreStatistics` RGxCC times i.e. we are going to call `.size` and `.computeIfAbsent`(hash + equals) that many times which is still not ideal. The cleaner fix would be at the caller side in `ParquetMetadataConverter.fromParquetMetadata()`: compute `shouldIgnoreStatistics` once before the row group loop using the file-level `createdBy`, then pass the pre-computed boolean through `buildColumnChunkMetaData` → `fromParquetStatisticsInternal`. This eliminates all per-column overhead — no hash, no lookup, just a boolean flowing through the call chain. This eliminates the need to check `shouldIgnoreStatistics` for every RGxCC. Both approaches could coexist (cache helps other callers), but the caller-side fix is where we see real improvements -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
