mzabaluev commented on PR #9700: URL: https://github.com/apache/arrow-rs/pull/9700#issuecomment-4245682722
> I wonder if this could be adapted to take a larger sample before deciding. Maybe raise the number of values per page if enabled. Can this be effected with other existing properties by an educated programmer? There may be adverse effects to tweaking other defaults dependently on this. If you mean `data_page_row_count_limit`, this is a writer property, so I'd be hesitant to auto-adjust it based on any column properties. I'd rather have the option to choose between: 1. The default simple behavior, i.e. `OnPageSizeLimit`; 2. The Java-like heuristic to closely follow Spark and other systems using parquet-java (however, a bit-for-bit workalike would require a larger rework of the logic and is perhaps not desirable); 3. Better adaptive behaviors, if and when implemented. For 3, I have purposefully left the `DictionaryFallback` enum open-ended. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
