Gabor, You're right that working with logical type data isn't really something that Parquet has done much in the past. We have the distinction between logical types and physical types to simplify what we need to support -- we only have encodings for physical types -- but in this case, we can't do that, and will need to use comparators based on the logical types. For now, we don't need to expose much to the layer above because all of the types have a defined sort order. In the future, we should be able to support other orders for UTF8 data, but we don't need to focus on that now.
Zoltan, While it isn't ideal to have no control over the sort order, the only thing we can do at the Parquet level is to handle data correctly when it does come in sorted. The concerns you're raising are good to think about, but we need to solve them elsewhere. I think that the table format should have a way to specify the sort order it wants for incoming data and communicate that to the engine writing Parquet files. That's what we're working on adding to the data source v2 API in Spark. Tables should be able to specify the expected clustering (for partitioning) and sort order for rows, then the query plan is automatically rewritten to make that happen. rb On Wed, Nov 8, 2017 at 7:47 AM, Zoltan Ivanfi <[email protected]> wrote: > Hi, > > I don't know the solution, just adding my thoughts. > > In my opinion the underlying problem is that min-max ranges have to be > small for the best filtering results. In order to achieve this, the data > has to be sorted, but calculating min-max statistics is the responsibility > of the library, while sorting the data is a responsibility of the > application above the library. Without logical-type-based sorting rules, > the application may sort data using a different ordering than the one used > for calculating/filtering on the min-max values. This results in too broad > min-max ranges, that in theory could still function correctly, but are not > optimal for filtering. (As an extra twist, even if we have > logical-type-based sorting rules, the application can still use a different > ordering for sorting and even for comparisons. The results can range from > overly broad min-max ranges to incorrectly discarded values.) > > From this point of view, dictionary entries and bloom filters are much less > problematic, because sorting by *any* order will optimize these structures, > as the only important thing is that equal values should end up next to each > other to increase the chance of having a low number of values in a single > page/row group. > > I think that trying to optimize min-max stats by sorting crosses the layer > boundary between the library and the application and as such is much better > suited to a full-stack implementation like Impala than the parquet-mr + > separate application stack. Neither relying on the application to calculate > standards-compliant statistics nor forcing the application to use the > library's data types for sorting and comparison seems like a good solution > to me. > > Please note that this problem inherently applies to columns statistics as > well. > > Zoltan > > On Wed, Nov 8, 2017 at 3:41 PM, Gabor Szadovszky < > [email protected]> wrote: > > > Hi, > > > > I started working on the jira PARQUET-1025 <https://issues.apache.org/ > > jira/browse/PARQUET-1025>. It is about implementing the new min-max > > statistics specified in PARQUET-686 <https://issues.apache.org/ > > jira/browse/PARQUET-686>. > > > > After looking in the code deeper I think the spec of the new min-max > stats > > contradicts to the actual design of parquet-mr: > > The concept of parquet-mr is to use the raw types (representing the > > primitive types in the spec) and let the client of the API have them > > converted to rich objects (representing the logical types). For example > > parquet-mr have Binary for both UTF8 and DECIMAL (at least for one > specific > > representation) instead of returning String or BigDecimal anywhere. > > The new min-max stats requires to have separate comparison mechanisms for > > the same primitives depending on the logical types. For example UTF8 > > requires unsigned (lexicographical) while DECIMAL requires signed > > comparisons for the same Binary class. > > The problem is that we are specifying sorting orders based on logical > > types while we are not providing specific java types for them. It means > the > > client can unintentionally use a different comparison logic than > parquet-mr > > in the min-max stats which can lead to discarding relevant values during > > filtering. > > > > I can see two possible solutions however none of them seems to be good > > enough for me. > > > > 1. Implement specific comparison logics based on the logical types in the > > related Statistics object > > The problem here is that we are still using the raw types therefore > client > > code in filters might implement different comparison logic than specified > > and implemented in Statistics. For example in case of having UINT_32 the > > min-max comparison in Statistics will use the proper unsigned comparison > > logic while the client code in case of checking the elements (in the > column > > or in the dictionary) might implement somewhat simpler e.g. value > 7 > which > > may lead to different results. See UserDefinedPredicate.keep(T) > > It is highly confusing for the client that it has to re-implement the > same > > comparison logic in the client code for the raw types as it is > implemented > > for the statistics. > > > > 2. Return specific “rich” objects for the logical types instead of the > raw > > types > > This solution would solve the problems what the previous one has but > would > > introduce bigger problems. > > Breaks the actual design of parquet-mr > > Backward incompatible: existing client code would not work properly for > > several logical types > > > > (3.) Revert the specs of the new min-max statistics > > Let’s keep working on the raw types without having any additional > > comparison logic. Specify the existing comparison logic (e.g. signed for > > all primitive types, (signed) lexicographical for binary) and expect the > > client to use these if it want to have sorting implemented for the values > > for better performance. > > > > > > What do you think, guys? Any suggestions on one of my solutions or for a > > new one I did not recognise? > > > > Thanks a lot, > > Gabor > > > -- Ryan Blue Software Engineer Netflix
