Hi, I started working on the jira PARQUET-1025 <https://issues.apache.org/jira/browse/PARQUET-1025>. It is about implementing the new min-max statistics specified in PARQUET-686 <https://issues.apache.org/jira/browse/PARQUET-686>.
After looking in the code deeper I think the spec of the new min-max stats contradicts to the actual design of parquet-mr: The concept of parquet-mr is to use the raw types (representing the primitive types in the spec) and let the client of the API have them converted to rich objects (representing the logical types). For example parquet-mr have Binary for both UTF8 and DECIMAL (at least for one specific representation) instead of returning String or BigDecimal anywhere. The new min-max stats requires to have separate comparison mechanisms for the same primitives depending on the logical types. For example UTF8 requires unsigned (lexicographical) while DECIMAL requires signed comparisons for the same Binary class. The problem is that we are specifying sorting orders based on logical types while we are not providing specific java types for them. It means the client can unintentionally use a different comparison logic than parquet-mr in the min-max stats which can lead to discarding relevant values during filtering. I can see two possible solutions however none of them seems to be good enough for me. 1. Implement specific comparison logics based on the logical types in the related Statistics object The problem here is that we are still using the raw types therefore client code in filters might implement different comparison logic than specified and implemented in Statistics. For example in case of having UINT_32 the min-max comparison in Statistics will use the proper unsigned comparison logic while the client code in case of checking the elements (in the column or in the dictionary) might implement somewhat simpler e.g. value > 7 which may lead to different results. See UserDefinedPredicate.keep(T) It is highly confusing for the client that it has to re-implement the same comparison logic in the client code for the raw types as it is implemented for the statistics. 2. Return specific “rich” objects for the logical types instead of the raw types This solution would solve the problems what the previous one has but would introduce bigger problems. Breaks the actual design of parquet-mr Backward incompatible: existing client code would not work properly for several logical types (3.) Revert the specs of the new min-max statistics Let’s keep working on the raw types without having any additional comparison logic. Specify the existing comparison logic (e.g. signed for all primitive types, (signed) lexicographical for binary) and expect the client to use these if it want to have sorting implemented for the values for better performance. What do you think, guys? Any suggestions on one of my solutions or for a new one I did not recognise? Thanks a lot, Gabor
