alamb commented on PR #10775: URL: https://github.com/apache/arrow-rs/pull/10775#issuecomment-5498015501
> I think I still need to be convinced it's the Parquet writer's job to figure out the best encoding options for a file, vs a user experimenting with their data to determine the same (a la https://github.com/XiangpengHao/parquet-linter). Yeah, this is my personal suggestion too - put more effort into organizing / teaching people how to choose the writer configuration. I think some sort of sampling library to auto tune parquet settings would be super helpful If there is any missing API to allow users control over how it works, then adding it t the parquet crate makes sense to me, but adding new policies here is less obvious to me. There are all sorts of tradeoffs picking heuristics (memory, cpu, etc) that I think any new policies that we added would need to demonstrate somehow a benefit to a large number of other users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
