JunRuiLee opened a new pull request, #10975:
URL: https://github.com/apache/paimon/pull/10975

   ### Purpose
   
   Enforce the existing VECTOR contract consistently at the Spark, Flink, and 
PyPaimon boundaries: a nullable vector may be NULL, but a non-NULL vector must 
not contain NULL elements.
   
   Spark/Flink array wrappers currently expose `[1.0, NULL, 3.0]` as a VECTOR, 
allowing writers such as Parquet to persist data that Paimon's binary/columnar 
vector readers reject. PyPaimon also accepts these elements in writes and 
legacy-file reads. Reject them with a clear error before exposing an engine 
array as a vector or passing Python inputs to physical writers. Spark V1 gets 
the same diagnostic instead of an unboxing exception.
   
   The Python checks cover Arrow and row writes (including Ray's datasink), 
row-ID/shard updates, native write/update/upsert bridges, and shared read 
adapters. Validation handles nested vectors, sliced/chunked Arrow arrays, and 
null parent containers. Whole-vector NULLs remain supported, ordinary ARRAY 
nullability is unchanged, and write validation respects selected columns and 
ignored RowKind events.
   
   ### Tests
   
   - Added 14 public Python write/read regression cases that fail on the base 
commit and pass with the fix.
   - Java: **19 passed** across `SparkVectorNullElementsTest`, 
`FlinkRowWrapperWithVectorTest`, `BinaryVectorTest`, and `ColumnarVecTest`:
     ```shell
     mvn -B -nsu -pl 
paimon-spark/paimon-spark-common,paimon-flink/paimon-flink-common -am 
-Pspark3,flink1 -DwildcardSuites=none -DfailIfNoTests=false 
-Dtest=SparkVectorNullElementsTest,FlinkRowWrapperWithVectorTest,BinaryVectorTest,ColumnarVecTest
 test
     ```
     Maven validation/checkstyle/spotless/enforcer checks were enabled.
   - Python: **102 passed, 12 skipped**, with Python 3.9 and PyArrow 19.0.1:
     ```shell
     PYTHONPATH=paimon-python python3 -m pytest 
paimon-python/pypaimon/tests/vector_null_elements_test.py 
paimon-python/pypaimon/tests/vector_table_test.py 
paimon-python/pypaimon/tests/parquet_vector_write_test.py 
paimon-python/pypaimon/tests/write/row_kind_write_test.py 
paimon-python/pypaimon/tests/write/ray_datasink_adapter_test.py -q
     ```
   - Flake8 passed for all changed Python files; `git diff --check` passed.
   
   Native boundary tests use mocked Rust delegates; the Rust runtime was not 
installed locally. The Ray regression invokes the real datasink locally without 
a Ray cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to