Zouxxyy opened a new issue, #11337: URL: https://github.com/apache/paimon/issues/11337
### Rules 1. **Use the statistics returned by the Parquet SDK as-is.** Preserve the reported bounds, counts, and missing values. Fix SDK write, read, or pruning bugs in the corresponding SDK and pick up the fix through a dependency upgrade. 2. **Use one comparison order for Paimon's own floating-point operations within each phase.** Use Java order during the transition and switch to `IEEE_754_TOTAL_ORDER` in the final state. This applies to Paimon-native predicates and the bounds Paimon computes and interprets. 3. **Prune only when the predicate cannot match.** Pushdown must preserve every row that can match the Paimon predicate. Track `null_count` and `nan_count` independently and treat a missing count as unknown. ### Transition Use Java `Float.compare` and `Double.compare` for comparisons Paimon controls. All NaNs compare equal and sort above positive infinity. Negative zero sorts before positive zero. #### Parquet SDKs For both FLOAT and DOUBLE, the sample `[5, NaN, NULL]` has `null_count=1` and no `nan_count` with the versions below. | SDK | Row-group bounds on write and reopen | Page index and SDK pruning | | --- | --- | --- | | parquet-java 1.16.0 | Writes `[5, NaN]`. Reopening under `TYPE_ORDER` drops the bounds and keeps the null count. | The sample has no column index for the floating-point column. A physical pruning reproducer on 1.16.0 passes with 1.18.2. Pick up that fix through the SDK upgrade. | | PyArrow 22.0.0 | Writes and reads `[5, 5]`. | The page bounds are also `[5, 5]` when the index is enabled. For `f != 5`, Arrow's in-memory filter returns the NaN row, while the Parquet Dataset scan can prune it. | | parquet-rs 59.3.2 | Writes and reads `[5, 5]`. | The page bounds are also `[5, 5]`. | For an all-NaN sample, parquet-java 1.16.0 writes `[NaN, NaN]`. PyArrow and parquet-rs omit the bounds. #### Paimon | Path | Java | Pure Python | Rust | | --- | --- | --- | --- | | Native predicates | Uses Java order. | Scalar predicates and Arrow expressions both evaluate `NaN > +Infinity` as false and treat signed zeros as equal. | Partition predicates use Java order. Data-column predicates use Arrow Rust, with negative NaNs below negative infinity and positive NaNs above positive infinity. | | Manifest-list partition statistics | Computes bounds in Java order. | Bounds on master depend on the input order. [#11317](https://github.com/apache/paimon/pull/11317) makes both `[5, NaN]` and `[NaN, 5]` produce `[5, NaN]`. | The writer uses Java order. For partitions `5` and a negative NaN, `p NOT BETWEEN 0 AND 10` can incorrectly skip the manifest. | | File-level key and value statistics | Regular writes use the writer's in-memory statistics. In a five-row-group sample, external file import reopens the footer and the SDK merge returns `[5, 5]`. Paimon then skips the file for `f > 5`. | The writer omits bounds for a column when it detects NaN and retains the null count. | Key stats come from the input using `partial_cmp`. Value stats come from the Parquet row groups. The one-row-group sample has `[5, 5]` and can be skipped for `f > 5`. | Paimon Rust also uses SDK metadata to select row groups and pages. Both Paimon selection paths have separate reproductions for dropping matching rows. ### Final state Use `IEEE_754_TOTAL_ORDER` for Paimon-native FLOAT and DOUBLE comparisons. Negative NaNs sort below negative infinity and positive NaNs sort above positive infinity. The order distinguishes NaN bit patterns and puts negative zero before positive zero. [Parquet column order](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1215-L1241) | SDK | Version or upstream work | | --- | --- | | Java | parquet-java **1.18.0** adds the total order and `nan_count` and defaults ordinary floating-point columns to that order. **1.18.2** was tested with bounds `[5, 5]` and `nan_count=1`. [1.18.0 release](https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.18.0) | | Rust | parquet-rs **60.0.0** adds the total order and read/write support for `nan_count`. `nan_count_opt()` distinguishes zero from missing. **60.2.0** is available. [Arrow Rust #9619](https://github.com/apache/arrow-rs/pull/9619) · [60.2.0 release](https://github.com/apache/arrow-rs/releases/tag/60.2.0) | | Python | PyArrow **26.0.1** lacks a `nan_count` API. **Arrow #50807** tracks C++ Parquet total order, NaN counts, and Dataset pruning. The Python statistics API and the row-level total-order comparison used by `Predicate.to_arrow()` need separate support. [PyArrow 26.0.1 statistics bindings](https://github.com/apache/arrow/blob/apache-arrow-26.0.1/python/pyarrow/_parquet.pyx#L64-L208) · [Arrow #50807](https://github.com/apache/arrow/pull/50807) | Use the SDK's `min/max` to check for a possible non-NaN match. When all non-null values are NaNs, interpret the bounds using the file's column order. Use `nan_count` to check for a possible NaN match. | `nan_count` | NaN pruning decision | | --- | --- | | `0` | Exclude NaNs from the candidate set. | | Positive | Retain the candidate if any possible NaN bit pattern satisfies the predicate. | | Missing | Treat NaNs as possible and apply the same candidate rule. | For example, `f > +Infinity` only matches positive NaNs under total order. A positive count gives only the total, so retain the candidate and let the row comparison check the sign. Verify new files with self-reads and cross-reads in Java, Python, and Rust. Cover signed NaNs with different bit patterns, signed zeros, infinities, all-NaN data, NaNs mixed with NULLs, multiple row groups, and multiple batches. The same predicate must return the same rows with pruning enabled and disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
