pitrou commented on PR #47086: URL: https://github.com/apache/arrow/pull/47086#issuecomment-3077248440
An issue here is that the change in default compression level will also apply to IPC buffer compression. The Arrow IPC format has a different usage model than Parquet. For Parquet, it is a reasonable decision to favor disk footprint and read performance. For Arrow IPC which is typically used for data streams (for example over Flight), write performance can be as important as read performance. Therefore, making writing an IPC stream 3x slower can be a big regression. Thoughts @lidavidm @paleolimbot @zeroshade ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
