Abhisheklearn12 commented on issue #11384: URL: https://github.com/apache/arrow-rs/issues/11384#issuecomment-6088655404
hey @andygrove, i'd like to pick this up if you haven't started on it. plan would be to port `DictionaryChooser` from comet (with credit to you ofc), but happy to back off if you'd rather do it yourself. @alamb re the goal: from what i can tell it's mostly faster selective reads, and smaller files as a side effect. with page indexes the reader always fetches the dictionary page of a column chunk, so unique columns (ids, uuids, timestamps) drag up to `dictionary_page_size_limit` along on every read even when only one page is selected. i ran a quick check on main: 1M rows in one row group, unique id/uuid/ts columns + a low cardinality one, default props vs dictionary turned off for just the unique columns (which is what this helper would return): - `id BETWEEN 1000 AND 1100`, zstd, all columns: 1.07 MB / 3.1 ms -> 0.29 MB / 0.56 ms - same thing uncompressed: 3.3 MB / 336 us -> 0.75 MB / 10 us - 100k row file (dictionary never hits the limit so it never falls back): 1.88 MB -> 1.36 MB - full scan: basically the same (39.9 vs 39.2 ms) synthetic data read from memory, so take it as rough. i used claude code to help put the bench together. tradeoffs as i see them: the writer itself doesn't change, so nobody is affected unless they call the helper. the real downside is what @etseidl pointed out, a small sample can pick plain for a medium cardinality column where the dictionary would have won. if the caller passes the sample, they can just send a bigger one when that matters. @mzabaluev on the api: i'd lean towards the caller passing the rows to sample (so sample size is theirs), keeping it dictionary only for now but naming it so other sample based tuning could go there later. i'd also make parquet-mr's first size check row a parameter instead of hardcoding 100, and handle the int96 / fixed len byte array leaves comet skips. does that sound like the right direction before i start? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
