alamb commented on issue #7723: URL: https://github.com/apache/arrow-rs/issues/7723#issuecomment-3025226729
That does sound like a good way to optimize dictionary encoding. One potential downside is that doing so it likely consume significantly more resources on write (niavely you would have to buffer the entire unencoded dataset, do the analysis on it, update the parquet writer settings, and then write it) However, I actually think it would be very interesting to try these high-resource requirement algorithms as a secondary "rewrite" / "optimization" pass -- most of the disaggregated database systems that work object store have some sort of compaction process anyways . And we can leverage the disaggregated architecture to do this heavyweight optimization on different machines than are taking reads or writes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
