discivigour opened a new pull request, #8650: URL: https://github.com/apache/paimon/pull/8650
## Purpose Schema evolution maps fields missing from an older data file to index `-1`. `VectorMappingUtils` previously represented those fields with a generic all-null `ColumnVector` lambda. Arrow complex-type writers select their implementation from the current logical schema and cast the runtime vector to `MapColumnVector`, `ArrayColumnVector`, or other typed vectors before writing, which can cause `ClassCastException` for missing complex columns. ## Changes - Add a singleton `AllNullColumnVector` to explicitly identify mapped columns whose values are all null. - Return the singleton for missing fields in `VectorMappingUtils.createMappedVectors`. - Short-circuit `ArrowFieldWriter.write` for the all-null singleton and empty batches before type-specific writers run. - Add coverage for complex all-null vectors, batch reuse, empty batches, missing-column mapping, and an end-to-end old-file/new-MAP-column schema evolution read. ## Tests - `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=VectorMappingUtilsTest,ArrowVectorizedBatchConverterTest,ArrowBatchConverterTest#testSchemaEvolutionWithMissingMapType test` - `mvn -pl paimon-common,paimon-arrow,paimon-core -DskipTests spotless:check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
