kerwin-zk opened a new pull request, #8456: URL: https://github.com/apache/paimon/pull/8456
### Purpose Since #6281, Paimon tables declare `ACCEPT_ANY_SCHEMA` for Spark v2 write, so Spark skips its own output resolution (`DataSourceV2Relation#skipSchemaResolution`) and the query is aligned to the table schema by `PaimonAnalysis` instead, which is only injected when `PaimonSparkSessionExtensions` is configured. If a write is planned without the extensions (e.g. `spark.paimon.requiredSparkConfsCheck.enabled=false` is set, or the plan is constructed by a third-party engine), a type-mismatched INSERT reaches the writer as-is: the writer interprets the incoming `UnsafeRow` with the table's row type, so a slot holding a DOUBLE gets read as a string's (offset, size). Depending on the bits this throws `NegativeArraySizeException` deep inside the format writer, crashes the executor with an out-of-bounds `Unsafe` read, or **silently writes garbage strings that can be committed** (we hit all three in a production case where a query wrote `SUM(...)` doubles into STRING columns). This PR adds a defense-in-depth fail-fast in `PaimonV2WriteBuilder#build`: the query schema must be positionally type-compatible with the table schema (reusing `DataType.equalsIgnoreCompatibleNullability`; CHAR/VARCHAR are normalized to STRING since they share the string binary layout). Mismatches now fail at planning time with a clear message pointing to `PaimonSparkSessionExtensions`. The check is skipped for row-level operations (`copyOnWriteScan` is set) and for `write.merge-schema` writes, which intentionally diverge from the current table schema and are validated elsewhere. ### Tests - New `PaimonV2WriteSchemaValidationTest`: without the extensions, a schema-matching INSERT still succeeds, while type-mismatched and column-count-mismatched INSERTs fail fast with the new error (previously the type mismatch failed at execution time with `NegativeArraySizeException`). - Regression: `PaimonOptionTest`, `PaimonConfigCheckTest`, `V2WriteMergeSchemaTest`, `V2WriteRequireDistributionTest`, `PaimonDynamicPartitionOverwriteCommandTest`, `SparkWriteITCase` pass on Spark 3.5 (Scala 2.12); `PaimonV2WriteSchemaValidationTest` + `V2WriteMergeSchemaTest` pass on Spark 4.x (Scala 2.13). ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
