JingsongLi commented on PR #10175:
URL: https://github.com/apache/paimon/pull/10175#issuecomment-5951915740
Reviewed head `6f4b683234542e086714d2e3c15218f356334d7f`. The read-plan
array filtering use case has end-to-end value, and the earlier non-ARRAY-field
validation finding is fixed. I found one additional numeric-array correctness
issue.
**[P2] Compare array elements using the declared element type**
(`predicate.py:516,535,553`). The new testers all use Python list membership,
but the builder accepts every `ArrayType` and discards the element type when
constructing the predicate. For a persisted `ARRAY<FLOAT>` written from
`[0.1]`, `array_contains("floats", 0.1)` returns no rows: the stored float32
becomes Python `0.10000000149011612`, while the literal remains a
double-precision Python float. `arrays_overlap` and `array_contains_all` have
the same false negative. There are also direct `ARRAY<DOUBLE>` differences from
Java's element comparator: a NaN literal misses a stored NaN, and either zero
literal matches both `-0.0` and `+0.0`, whereas Java `Float/Double.compareTo`
matches NaN and distinguishes signed zero.
I reproduced these through actual table writes, commits, reloads and reads
in Parquet, Avro, ORC and row formats, projecting only the ID column. Please
retain the element type, normalize literals accordingly, and use the same typed
comparison contract as Java; alternatively reject unsupported element types
explicitly. Add numeric-array tests for all three methods. This finding is
scoped to the newly accepted array-predicate paths, not the existing scalar
Python predicates.
Validation: 86 Python array/predicate/read/projection tests passed; 22 Java
PredicateBuilder tests passed on JDK 8 with normal Maven checks. Across all
four formats, 24 additional string-array controls passed for exact row IDs,
projection, compound OR, empty literal lists and nulls; the numeric mismatches
above reproduced in every format. Configured flake8, Python 3.6 grammar and
diff checks passed; current-head CI is green. The no-file-pruning limitation is
now accurately described in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
