Hi, I don't know the solution, just adding my thoughts.
In my opinion the underlying problem is that min-max ranges have to be small for the best filtering results. In order to achieve this, the data has to be sorted, but calculating min-max statistics is the responsibility of the library, while sorting the data is a responsibility of the application above the library. Without logical-type-based sorting rules, the application may sort data using a different ordering than the one used for calculating/filtering on the min-max values. This results in too broad min-max ranges, that in theory could still function correctly, but are not optimal for filtering. (As an extra twist, even if we have logical-type-based sorting rules, the application can still use a different ordering for sorting and even for comparisons. The results can range from overly broad min-max ranges to incorrectly discarded values.) >From this point of view, dictionary entries and bloom filters are much less problematic, because sorting by *any* order will optimize these structures, as the only important thing is that equal values should end up next to each other to increase the chance of having a low number of values in a single page/row group. I think that trying to optimize min-max stats by sorting crosses the layer boundary between the library and the application and as such is much better suited to a full-stack implementation like Impala than the parquet-mr + separate application stack. Neither relying on the application to calculate standards-compliant statistics nor forcing the application to use the library's data types for sorting and comparison seems like a good solution to me. Please note that this problem inherently applies to columns statistics as well. Zoltan On Wed, Nov 8, 2017 at 3:41 PM, Gabor Szadovszky < [email protected]> wrote: > Hi, > > I started working on the jira PARQUET-1025 <https://issues.apache.org/ > jira/browse/PARQUET-1025>. It is about implementing the new min-max > statistics specified in PARQUET-686 <https://issues.apache.org/ > jira/browse/PARQUET-686>. > > After looking in the code deeper I think the spec of the new min-max stats > contradicts to the actual design of parquet-mr: > The concept of parquet-mr is to use the raw types (representing the > primitive types in the spec) and let the client of the API have them > converted to rich objects (representing the logical types). For example > parquet-mr have Binary for both UTF8 and DECIMAL (at least for one specific > representation) instead of returning String or BigDecimal anywhere. > The new min-max stats requires to have separate comparison mechanisms for > the same primitives depending on the logical types. For example UTF8 > requires unsigned (lexicographical) while DECIMAL requires signed > comparisons for the same Binary class. > The problem is that we are specifying sorting orders based on logical > types while we are not providing specific java types for them. It means the > client can unintentionally use a different comparison logic than parquet-mr > in the min-max stats which can lead to discarding relevant values during > filtering. > > I can see two possible solutions however none of them seems to be good > enough for me. > > 1. Implement specific comparison logics based on the logical types in the > related Statistics object > The problem here is that we are still using the raw types therefore client > code in filters might implement different comparison logic than specified > and implemented in Statistics. For example in case of having UINT_32 the > min-max comparison in Statistics will use the proper unsigned comparison > logic while the client code in case of checking the elements (in the column > or in the dictionary) might implement somewhat simpler e.g. value > 7 which > may lead to different results. See UserDefinedPredicate.keep(T) > It is highly confusing for the client that it has to re-implement the same > comparison logic in the client code for the raw types as it is implemented > for the statistics. > > 2. Return specific “rich” objects for the logical types instead of the raw > types > This solution would solve the problems what the previous one has but would > introduce bigger problems. > Breaks the actual design of parquet-mr > Backward incompatible: existing client code would not work properly for > several logical types > > (3.) Revert the specs of the new min-max statistics > Let’s keep working on the raw types without having any additional > comparison logic. Specify the existing comparison logic (e.g. signed for > all primitive types, (signed) lexicographical for binary) and expect the > client to use these if it want to have sorting implemented for the values > for better performance. > > > What do you think, guys? Any suggestions on one of my solutions or for a > new one I did not recognise? > > Thanks a lot, > Gabor >
