alamb commented on issue #10489: URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5182744600
Thank you for the report @jonasdedden > When an individual value is larger than data_page_size_limit, the chunker is forced to emit one value per page. I think we could potentially argue this is an **improvement** in behavior in the sense that the output data pages are now closer to the specified limit. I realize that for the usecase described in this ticket the compression ratio is likely much worse, but I would argue that may have been because prior versions weren't accurately applying the page size limit. <img width="1123" height="584" alt="Image" src="https://github.com/user-attachments/assets/456bb9ea-9d8e-4c9b-b7da-4a4baa7ea396"; /> It seems like the correct fix for the issue you are seeing would be to increase your page size limit to be larger than the values you are writing into the encoder, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
