tustvold commented on issue #5037: URL: https://github.com/apache/arrow-rs/issues/5037#issuecomment-1793858520
Hmm... I also note that it is disabled by default, is this still the case? Regardless I think we should probably only perform this in the context of https://github.com/apache/parquet-format/pull/216 as whilst parquet-mr would appear to be configurable to perform binary truncation, I'm fairly confident there are applications that have implicit assumptions that this would break. FYI @alamb my memory is hazy as to what forms of aggregate pushdown DF performs, and if we might need to introduce some notion of inexact statistics (if it doesn't already exist). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
