JingsongLi commented on PR #10173: URL: https://github.com/apache/paimon/pull/10173#issuecomment-5966588235
Reviewed current head `35480bcf5e79df26e0e688fe26570c0e41440d1d`. The previous NaN and nanosecond timestamp findings are fixed: actual Python-written Avro/Parquet tables now retain their matching rows in Java scans under `truncate(3)`, including real +/-1ns and +/-2ns values. The new unreliable-bound flag also survives a real multi-row-group row-ID update; replacing just the previous incremental writer function reproduces the nanosecond pruning failure. One remaining production correctness issue affects the newly enabled row-ID-update statistics: **[P1] Preserve signed-zero bounds when merging update-file batches** (`paimon-python/pypaimon/write/table_update_by_row_id.py:150-155,168-171`, `write/writer/single_file_writer.py:75-81`). A row-ID partial update can write one Parquet file incrementally over several batches. `SingleFileWriter` merges each batch's min/max using Python `min`/`max`, which treat `-0.0` and `+0.0` as equal and retain the first value. With negative zero in the first batch and positive zero in the second, the file therefore publishes min=max `-0.0` even though it physically contains `+0.0`. Reversing the batches instead publishes min=max `+0.0`, losing the lower negative-zero bound. Neither column contains NaN or a nanosecond timestamp, so the new omission flag stays false. I reproduced both orders through the public write/update/commit/reload APIs on real row-tracking/data-evolution tables, with `metadata.stats-mode=truncate(3)` and the supported `file.block-size='1 b'` setting to make the two-row example produce two row groups. Larger updates crossing the ordinary row-group limit take the same incremental path; no metadata was fabricated. Java unfiltered reads preserve both zero bit patterns. On the negative-first table, Java `equal(d, +0.0)` and `greaterThan(d, -0.0)` both plan zero splits and return nothing instead of ID 1. On the positive-first table, `equal(d, -0.0)` and `lessThan(d, +0.0)` likewise lose ID 1. The committed delta manifest advertises the incorrect one-column bounds and null count 0. Identical public update flows under `none` and `counts` pass all 20 Java query controls, so this witness does not depend on a Parquet row-group reader failure. The shared incremental merger already had this limitation under `full`; this finding concerns the PR enabling those bounds for the existing `truncate(N)` option on the row-ID-update path. The original PR baseline collected this path's fields only for `full`, so `truncate(N)` previously emitted no bounds and could not make this pruning decision. Ordinary single-batch Arrow min/max handles signed zero correctly and is not the failing route. Please merge floating bounds using the Java typed ordering, or conservatively omit them when that ordering cannot be guaranteed, and add a persisted multi-batch row-ID-update regression. Validation: 126 Python tests passed (22 native-extension tests skipped locally); 12 Java collector tests passed with JDK 8 and normal Maven checks. Configured flake8/Python 3.6 grammar checks pass; current-head CI is green. The additional Python-to-Java tests cover 28 committed tables across mode/type controls, plus real incremental row-ID updates and the old-function controls described above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
