wgtmac commented on PR #3393: URL: https://github.com/apache/parquet-java/pull/3393#issuecomment-5125834732
Digging into this further, I found that old parquet-java writers did write NaN into the bounds, despite the spec guidance. Old readers then discarded the min/max whenever either bound was NaN, which made this safe in practice. The new writer changes that behavior for `TYPE_ORDER`: it writes finite bounds computed from the non-NaN values together with nan_count. An old reader ignores `nan_count` but accepts the finite bounds. This can lead to incorrect pruning for predicates such as `eq(x, NaN)`, `in(... NaN ...)`, `notEq(x, finite)`, and `gt/gtEq(x, finite)`. For example, `[1.0, NaN]` now produces `min = max = 1.0`. An old parquet-java reader may drop the row group for `gt(x, 1.0)`, even though NaN satisfies that predicate according to its comparator. Strictly speaking, the old readers have a latent bug because they are not conservative enough for spec-conforming `TYPE_ORDER` statistics. However, this change exposes that bug for files produced by parquet-java, so I agree that it is also a backward-compatibility issue for the new writer. I'm not sure if it is worth fixing provided that the backward-compatible behavior does not look right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
