mzabaluev commented on PR #9700: URL: https://github.com/apache/arrow-rs/pull/9700#issuecomment-4334656223
Cleanly outperforming the dictionary-disabled setting would be more challenging due to the eager dumping of dictionary-encoded pages, as described in #9739. And since the check is performed on write batch granularity, it may be too late to recover regardless, unless the whole encoder is rewritten to buffer and re-encode from the beginning of the column chunk. But this is fallback behavior to catch bad cases, not to provide best case optimized behavior. As stated in the above discussion, the user is ultimately responsible for fine-tuning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
