alamb opened a new pull request #643: URL: https://github.com/apache/arrow-rs/pull/643
# Which issue does this PR close? Closes https://github.com/apache/arrow-rs/issues/641 # Rationale for this change Statistics for strings (aka ByteArrays in the parquet format) are not calculated correctly For example, given the strings "z", and "aa", the parquet writer will determine that "z" is the *minimum* value (because "z" is shorter). # What changes are included in this PR? 1. Fix the comparison code for ByteArray to lexographically compare the data 2. Add more statistics test coverage (kudos to @crepererum for the test harness in https://github.com/apache/arrow-rs/pull/256 which made this easy) ## Correctness I verified that the python implementation compares the strings lexicographically as proposed in this PR rather than the current behavior (see https://github.com/apache/arrow-rs/issues/641#issuecomment-890325580) I also messed around with how partial cmp works with `std::Option` and I believe this implementation is consistent. You can see it for yourself in [this playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=418adb826f28d7f11d725ccaf8d770ba) ## Notes I could not figure out how to test for "Null" values in statistics (though I tested end to end using the ArrowWriter and confirmed null values didn't appear in the statistics, as expected) Also, there are three things testing revealed: 1. BoolType columns produce Int32Stats which I found confusing 2. FixedLenByteArray columns produce ByteArrayStats 3. I don't know enough about how Int96 works to properly test it (my attempt failed badly) # Are there any user-facing changes? correct statistics -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
