etseidl commented on issue #11384: URL: https://github.com/apache/arrow-rs/issues/11384#issuecomment-6017999424
My issue from the start has been that the java behavior is a holdover from a time when the default page size was 1MB (so 256k int32 values, 128k int64, etc). Post page index introduction, pages were limited to 20_000 rows, a much smaller sample size. So a column with a moderate cardinality in the 30k range will fallback to plain encoding, even though the dictionary would actually help a great deal. I think users are in a much better position to know the cardinality of their data than low-level parquet encoding engines with tiny samples. As long as the engine is configurable, I don't see an issue with placing the burden of tuning on the user (or downstream database layer). Solutions proposed for this to date, IMO, have suffered from adding complexity for very little real gain. Certainly no gain over simply setting dictionary encoding to `false` for high cardinality columns. > A function in the parquet crate that takes the Arrow schema, the WriterProperties and the first rows to be written, replays parquet-java's first-page accounting for each leaf column, and returns the properties with set_column_dictionary_enabled(path, false) for every column parquet-java would write plain. This I would support. It moves the guessing game from the encoder to a separate API that behaves as a "linter" (like https://github.com/XiangpengHao/parquet-linter). The downstream user then decides the sample size and lives with the consequences...want a better estimate, send a larger sample. I agree with @mzabaluev that it should have the ability to decide more than just dictionary true/false. But we could start with that and add more capability later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
