jackylee-ch opened a new pull request, #9014:
URL: https://github.com/apache/paimon/pull/9014
### Purpose
`SparkCatalog.toInitialSchema` converts a column listed in the
`vector-field` option into a Paimon `VECTOR`. Three problems on that path:
1. The array check reports `"The type of blob field must be array"` — the
message was copied from the BLOB branch above it and names the wrong option.
2. `checkArgument` is called with a template containing `%s` but no varargs,
so `Preconditions` leaves the placeholders unsubstituted and the user is told
to set the literal key `field.%s.vector-dim`.
`LogicalTypeConversion.toVectorType` on the Flink side has the same defect.
3. `Integer.parseInt(properties.get(dimKey))` is unguarded, so a non-integer
dimension surfaces as a raw `NumberFormatException: For input string: "abc"`
from inside the catalog. Flink already guards this; Spark did not.
`DataTypes.VECTOR(length, element)` also forces `isNullable = true`, so a
`NOT NULL` array column declared in Spark silently lost its nullability. The
BLOB branch in the same method and the `__VECTOR_FIELD` comment-directive path
in `ColumnDirectiveUtils.convertType` both preserve it.
### Changes
- Extract the vector conversion into `SparkCatalog#toVectorType`, mirroring
the existing `toBlobType`: name the option correctly, pass the `checkArgument`
arguments, reject blank/non-integer dimensions with the option key in the
message, and build `VectorType` with the declared nullability.
- Align the two `checkArgument` messages in Flink's `toVectorType` and reuse
`StringUtils.isNullOrWhitespaceOnly` so a whitespace-only dimension is rejected
the same way in both engines.
No storage format or option semantics change; only validation messages and
the nullability carried into the schema.
### Tests
- Added `Paimon DDL: create table with vector-field` and `Paimon DDL: create
table with invalid vector-field` to `DDLTestBase`, covering the round-trip
(length, element type, `NOT NULL`) and all four rejection paths. These run for
every Spark version via the per-module `DDLTest` subclasses.
```
mvn -pl paimon-spark/paimon-spark-3.5 -am -Pfast-build,spark3
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.DDLTest -Dtest=none test
-> 35 succeeded, 0 failed (33 on master before this change)
mvn -pl paimon-flink/paimon-flink-common -Pfast-build
-Dtest=LogicalTypeConversionTest test
-> 6 succeeded, 0 failed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
