etseidl commented on PR #10775: URL: https://github.com/apache/arrow-rs/pull/10775#issuecomment-5496456535
I finally got around to testing this with my dict breaker dataset. This is some fake HR data with 200M rows. One column, "Address", has a cardinality of 80000, but fairly large records, so the 1MB dictionary limit is hit fairly soon (after about 33k rows). So this sits in a pretty diabolical place: by the time a decision wants to be made, there aren't really enough samples available to make a firm determination as to whether the dictionary is buying you anything. I added some prints at fallback time, and found that when the 1MB limit is hit, the dictionary size is `1049059` and the plain encoded size is `1368926`. The encoded page sizes at this point add up to another 62k, so total cost for dict is arund 1.1MB. With the current behavior, fallback will occur, and result in a 7GB file. If I crank up the dict size limit to 4MB, then fallback never occurs and the file size is 2.5GB. I tested this with the new `WhenProfitable` strategy, and found that premature fallback will still occur until I crank the ratio over 0.7. So to sum up, I guess I'm still not seeing what the benefit is here. I can currently set the dict size limit for the problematic column to something large enough, or with this change I can instead set a different strategy after some testing to get the same result. In either case, I still need to know my data and tune appropriately if I want to prioritize file size over dictionary size. I think I still need to be convinced it's the Parquet writer's job to figure out the best encoding options for a file, vs a user experimenting with their data to determine the same (a la https://github.com/XiangpengHao/parquet-linter). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
