Zouxxyy opened a new issue, #11337:
URL: https://github.com/apache/paimon/issues/11337

   ### Rules
   
   1. **Use the statistics returned by the Parquet SDK as-is.** Preserve the 
reported bounds, counts, and missing values. Fix SDK write, read, or pruning 
bugs in the corresponding SDK and pick up the fix through a dependency upgrade.
   2. **Use one comparison order for Paimon's own floating-point operations 
within each phase.** Use Java order during the transition and switch to 
`IEEE_754_TOTAL_ORDER` in the final state. This applies to Paimon-native 
predicates and the bounds Paimon computes and interprets.
   3. **Prune only when the predicate cannot match.** Pushdown must preserve 
every row that can match the Paimon predicate. Track `null_count` and 
`nan_count` independently and treat a missing count as unknown.
   
   ### Transition
   
   Use Java `Float.compare` and `Double.compare` for comparisons Paimon 
controls. All NaNs compare equal and sort above positive infinity. Negative 
zero sorts before positive zero.
   
   #### Parquet SDKs
   
   For both FLOAT and DOUBLE, the sample `[5, NaN, NULL]` has `null_count=1` 
and no `nan_count` with the versions below.
   
   | SDK | Row-group bounds on write and reopen | Page index and SDK pruning |
   | --- | --- | --- |
   | parquet-java 1.16.0 | Writes `[5, NaN]`. Reopening under `TYPE_ORDER` 
drops the bounds and keeps the null count. | The sample has no column index for 
the floating-point column. A physical pruning reproducer on 1.16.0 passes with 
1.18.2. Pick up that fix through the SDK upgrade. |
   | PyArrow 22.0.0 | Writes and reads `[5, 5]`. | The page bounds are also 
`[5, 5]` when the index is enabled. For `f != 5`, Arrow's in-memory filter 
returns the NaN row, while the Parquet Dataset scan can prune it. |
   | parquet-rs 59.3.2 | Writes and reads `[5, 5]`. | The page bounds are also 
`[5, 5]`. |
   
   For an all-NaN sample, parquet-java 1.16.0 writes `[NaN, NaN]`. PyArrow and 
parquet-rs omit the bounds.
   
   #### Paimon
   
   | Path | Java | Pure Python | Rust |
   | --- | --- | --- | --- |
   | Native predicates | Uses Java order. | Scalar predicates and Arrow 
expressions both evaluate `NaN > +Infinity` as false and treat signed zeros as 
equal. | Partition predicates use Java order. Data-column predicates use Arrow 
Rust, with negative NaNs below negative infinity and positive NaNs above 
positive infinity. |
   | Manifest-list partition statistics | Computes bounds in Java order. | 
Bounds on master depend on the input order. 
[#11317](https://github.com/apache/paimon/pull/11317) makes both `[5, NaN]` and 
`[NaN, 5]` produce `[5, NaN]`. | The writer uses Java order. For partitions `5` 
and a negative NaN, `p NOT BETWEEN 0 AND 10` can incorrectly skip the manifest. 
|
   | File-level key and value statistics | Regular writes use the writer's 
in-memory statistics. In a five-row-group sample, external file import reopens 
the footer and the SDK merge returns `[5, 5]`. Paimon then skips the file for 
`f > 5`. | The writer omits bounds for a column when it detects NaN and retains 
the null count. | Key stats come from the input using `partial_cmp`. Value 
stats come from the Parquet row groups. The one-row-group sample has `[5, 5]` 
and can be skipped for `f > 5`. |
   
   Paimon Rust also uses SDK metadata to select row groups and pages. Both 
Paimon selection paths have separate reproductions for dropping matching rows.
   
   ### Final state
   
   Use `IEEE_754_TOTAL_ORDER` for Paimon-native FLOAT and DOUBLE comparisons. 
Negative NaNs sort below negative infinity and positive NaNs sort above 
positive infinity. The order distinguishes NaN bit patterns and puts negative 
zero before positive zero. [Parquet column 
order](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1215-L1241)
   
   | SDK | Version or upstream work |
   | --- | --- |
   | Java | parquet-java **1.18.0** adds the total order and `nan_count` and 
defaults ordinary floating-point columns to that order. **1.18.2** was tested 
with bounds `[5, 5]` and `nan_count=1`. [1.18.0 
release](https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.18.0)
 |
   | Rust | parquet-rs **60.0.0** adds the total order and read/write support 
for `nan_count`. `nan_count_opt()` distinguishes zero from missing. **60.2.0** 
is available. [Arrow Rust #9619](https://github.com/apache/arrow-rs/pull/9619) 
· [60.2.0 release](https://github.com/apache/arrow-rs/releases/tag/60.2.0) |
   | Python | PyArrow **26.0.1** lacks a `nan_count` API. **Arrow #50807** 
tracks C++ Parquet total order, NaN counts, and Dataset pruning. The Python 
statistics API and the row-level total-order comparison used by 
`Predicate.to_arrow()` need separate support. [PyArrow 26.0.1 statistics 
bindings](https://github.com/apache/arrow/blob/apache-arrow-26.0.1/python/pyarrow/_parquet.pyx#L64-L208)
 · [Arrow #50807](https://github.com/apache/arrow/pull/50807) |
   
   Use the SDK's `min/max` to check for a possible non-NaN match. When all 
non-null values are NaNs, interpret the bounds using the file's column order. 
Use `nan_count` to check for a possible NaN match.
   
   | `nan_count` | NaN pruning decision |
   | --- | --- |
   | `0` | Exclude NaNs from the candidate set. |
   | Positive | Retain the candidate if any possible NaN bit pattern satisfies 
the predicate. |
   | Missing | Treat NaNs as possible and apply the same candidate rule. |
   
   For example, `f > +Infinity` only matches positive NaNs under total order. A 
positive count gives only the total, so retain the candidate and let the row 
comparison check the sign.
   
   Verify new files with self-reads and cross-reads in Java, Python, and Rust. 
Cover signed NaNs with different bit patterns, signed zeros, infinities, 
all-NaN data, NaNs mixed with NULLs, multiple row groups, and multiple batches. 
The same predicate must return the same rows with pruning enabled and disabled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to