I don't think reverting is the right direction. The spec is explicit about the required reader behavior for TYPE_ORDER:
> When looking for NaN values, min and max should be ignored. Old parquet-java readers are not conservative enough under these rules. The current writer combination of TYPE_ORDER, finite non-NaN bounds, and nan_count can expose that legacy reader bug. However, I don't think preserving the old incorrect behavior or reverting the feature is the right solution. This PR exactly fixes that behavior. This is the problem the new order was introduced to solve. My preference is to keep the format and reader changes, enable IEEE_754_TOTAL_ORDER in the writer. It tells legacy readers that they do not understand the ordering and should ignore the statistics, while new readers can use the total order and nan_count correctly. On Thu, Jul 30, 2026 at 4:27 PM Divjot Arora via dev <[email protected]> wrote: > Hi folks, > > I'm starting this thread to discuss how we add and enable new sort orders > in the format. There was a recent change [1] to add a new > IEEE_754_TOTAL_ORDER sort order and a nan_count field to row-group/page > stats. TLDR for this change: NaN values invalidate stats, so writers leave > them out of min/max stats and instead set nan_count to signal whether NaN's > are present. > > The merged parquet-java implementation [2] emits nan_count from the writer, > but not the new sort order (this is valid per the spec). The arrow-rs > implementation [3] emits both nan_count and the sort order, but is not > merged and is marked with "api-release" and "next-major-release" labels. > There seems to be a disconnect between the format spec and the > implementations: the change went into to parquet-format v2.13.0 and was > considered forward-compatible, but both reference implementations are > considering use of the new sort order to be forward-incompatible. > Additionally, adopting nan_count but not IEEE_754_TOTAL_ORDER can cause > correctness as described parquet-java PR [4]. > > Given that the writer changes will go out in parquet-java 1.18 and there is > another new sort order (INT96_TIMESTAMP_ORDER) merged but not yet released > in parquet-format, we need to decide how to handle these cases. I see two > options: > > 1. Consider new sort orders to be forward-compatible and update the > implementations to adopt IEEE_754_TOTAL_ORDER. Per the spec, readers should > gracefully handle unrecognized sort orders by ignoring min/max stats [5]. > This is the simplest approach, but risks older readers breaking. Per > discussion on the initial nan_count thread [6] the Java, C++, and Python > implementations have been verified to gracefully ignore unrecognized union > values. Older versions of arrow-rs would fail, but this has been fixed in > newer versions. > 2. Consider new sort orders to be forward-incompatible at the format level. > Revert the IEEE_754_TOTAL_ORDER/nan_count change as well as the > INT96_TIMESTAMP_ORDER in advance of parquet-format v2.14.0. Revert the > parquet-java reference implementation before releasing v1.18 as well. This > is the safest approach, but it would mean both new sort orders would be > blocked by the ongoing versioning discussion and released only in the next > format major version. > > Given past discussion and verification of different implementations, my > opinion is that this is largely a forward compatible change. Regardless of > what we decide, this discussion should block the ongoing parquet-java > release as the current state (emitting stats with TYPE_DEFINED_ORDER and > nan_count) is not forward compatible and poses correctness risks. > > -- Div > > [1] > > https://github.com/apache/parquet-format/commit/662cdac7ffb7e71e10e0e0c519b215791ff2d1aa > [2] https://github.com/apache/parquet-java/pull/3393 > [3] https://github.com/apache/arrow-rs/pull/9619 > [4] > https://github.com/apache/parquet-java/pull/3393#issuecomment-5125834732 > [5] > > https://github.com/apache/parquet-format/blob/2076361bb64e2de9ca6a8d06eda025a6fa4e9df6/src/main/thrift/parquet.thrift#L1100 > [6] https://lists.apache.org/thread/t93kw60gkn3lq7vt4n6c4nkmsr7dtyn1 >
