princejain756 commented on PR #11904: URL: https://github.com/apache/arrow-rs/pull/11904#issuecomment-6542199360
Sorry for the slow follow-up @Rich-T-kid, that one is on me. On the regression: you're right. The gather + remap pass costs a few microseconds per batch, so on small or dense dictionaries the compaction is pure overhead. `values.len() > dictionary.len()` is too loose a gate. Something like `values.len() > 8 * dictionary.len() && values.len() > 65536` keeps today's path for those cases and still catches the 1M-values/8k-rows ClickBench workload. If the direction is still interesting I can add the small-dict benchmark cases you listed and retest before reopening. For the record on the AI-spam flag: the change was AI-assisted, with the benchmark setup, tests, and the encode/remap split verified by hand. The missed review window was a human mistake, not a missing human. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
