JunRuiLee opened a new pull request, #10975:
URL: https://github.com/apache/paimon/pull/10975
### Purpose
Enforce the existing VECTOR contract consistently at the Spark, Flink, and
PyPaimon boundaries: a nullable vector may be NULL, but a non-NULL vector must
not contain NULL elements.
Spark/Flink array wrappers currently expose `[1.0, NULL, 3.0]` as a VECTOR,
allowing writers such as Parquet to persist data that Paimon's binary/columnar
vector readers reject. PyPaimon also accepts these elements in writes and
legacy-file reads. Reject them with a clear error before exposing an engine
array as a vector or passing Python inputs to physical writers. Spark V1 gets
the same diagnostic instead of an unboxing exception.
The Python checks cover Arrow and row writes (including Ray's datasink),
row-ID/shard updates, native write/update/upsert bridges, and shared read
adapters. Validation handles nested vectors, sliced/chunked Arrow arrays, and
null parent containers. Whole-vector NULLs remain supported, ordinary ARRAY
nullability is unchanged, and write validation respects selected columns and
ignored RowKind events.
### Tests
- Added 14 public Python write/read regression cases that fail on the base
commit and pass with the fix.
- Java: **19 passed** across `SparkVectorNullElementsTest`,
`FlinkRowWrapperWithVectorTest`, `BinaryVectorTest`, and `ColumnarVecTest`:
```shell
mvn -B -nsu -pl
paimon-spark/paimon-spark-common,paimon-flink/paimon-flink-common -am
-Pspark3,flink1 -DwildcardSuites=none -DfailIfNoTests=false
-Dtest=SparkVectorNullElementsTest,FlinkRowWrapperWithVectorTest,BinaryVectorTest,ColumnarVecTest
test
```
Maven validation/checkstyle/spotless/enforcer checks were enabled.
- Python: **102 passed, 12 skipped**, with Python 3.9 and PyArrow 19.0.1:
```shell
PYTHONPATH=paimon-python python3 -m pytest
paimon-python/pypaimon/tests/vector_null_elements_test.py
paimon-python/pypaimon/tests/vector_table_test.py
paimon-python/pypaimon/tests/parquet_vector_write_test.py
paimon-python/pypaimon/tests/write/row_kind_write_test.py
paimon-python/pypaimon/tests/write/ray_datasink_adapter_test.py -q
```
- Flake8 passed for all changed Python files; `git diff --check` passed.
Native boundary tests use mocked Rust delegates; the Rust runtime was not
installed locally. The Ray regression invokes the real datasink locally without
a Ray cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]