MashrafeeAryan opened a new pull request, #52579: URL: https://github.com/apache/arrow/pull/52579
Title: GH-47752: [C++][Python][Parquet] Add minimum compression savings for V2 pages Fixes #47752. ### Rationale for this change Parquet's V2 pages currently keep compression whenever the compressed values are smaller than the original values. Even if it saves very little space, it still does it. For instance, it can compress 100 KB to 99 KB and save it. Even though that's 1% storage efficient, it might not be worth the decompression overhead while reading the file. ### What changes are included in this PR? In the PR, I added an optional `min_space_savings` setting between `0.0` and `1.0`. It gives users an option to select the minimum threshold to compress V2 page data values. For instance, if the original file is `100 KB` and the user has `min_space_savings = 0.1`, it means the compressed file needs to save at least 10% space. As a result, unless the file is `<= 90 KB`, it won't get saved. I keep the current behavior when the user does not set `min_space_savings`. Added C++ and Python implementation. ### Are these changes tested? - C++ implementation has been tested. - Working on Python implementation testing. This is a draft PR. Will do more tests. ### Are there any user-facing changes? C++ and Python users can set `min_space_savings` when writing V2 pages. For example, `0.1` requires at least 10% space savings before keeping compression. ### Was AI used for this PR? Codex helped navigate and suggest changes for tests and compression behavior. Claude helped to review changes and point out any critical bugs. **PR code and description written by:** - [x] Human (I wrote it. Used ChatGPT to fix my grammar mistakes.) - [ ] AI **Reviewed before submission by:** - [x] Human - [ ] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
