mdibaiee opened a new pull request, #3575: URL: https://github.com/apache/parquet-java/pull/3575
<!-- Thanks for opening a pull request! If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose and format pull request title as below: GH-${GITHUB_ISSUE_ID}: ${SUMMARY} or simply use the title below if it is a minor issue: MINOR: ${SUMMARY} --> ### Rationale for this change Missing `null_count` statistics for columns in parquet files can cause issues with downstream consumers of these files. It is not necessary to omit this statistic for columns which are larger than the truncation configuration, since despite the truncation, their nullability can be asserted with confidence. It is reasonable to keep omitting min/max statistics due to the rationale explained in the comment in the code. ### What changes are included in this PR? Always add `null_count` statistics for columns in parquet files, unconditional of their size. ### Are these changes tested? Yes, `TestParquetMetadataConverter.java` has been updated to reflect these changes ### Are there any user-facing changes? I think we can consider the additional `null_count` statistic's appearance as a user-facing change <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. --> Closes #3574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
