JingsongLi commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3523236194
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2WriteBuilder.scala:
##########
@@ -55,4 +61,63 @@ class PaimonV2WriteBuilder(table: FileStoreTable,
dataSchema: StructType, option
}
override def partitionRowType(): RowType =
table.schema().logicalPartitionType()
+
+ /**
+ * Fail fast when the query schema is binary-incompatible with the table
schema.
+ *
+ * Paimon tables declare `ACCEPT_ANY_SCHEMA`, so Spark skips its own output
resolution and the
+ * query is aligned to the table schema by `PaimonAnalysis` instead, which
is only injected when
+ * `PaimonSparkSessionExtensions` is configured. If a write is planned
without it, a
+ * type-mismatched query would reach the writer as-is and be interpreted
with the table's row
+ * type, silently corrupting data or failing with errors like
`NegativeArraySizeException` deep
+ * inside the format writer.
+ */
+ private def validateDataSchema(): Unit = {
+ // Row-level operations write the table's own rows back; schema evolution
intentionally
+ // diverges from the current table schema and is validated when merging.
+ if (copyOnWriteScan.nonEmpty ||
SchemaEvolutionHelper.mergeSchemaEnabled(options)) {
+ return
+ }
+
+ val expectedFields =
SparkTypeUtils.fromPaimonRowType(table.rowType()).fields
+ val actualFields =
SparkSystemColumns.filterSparkSystemColumns(dataSchema).fields
Review Comment:
This does not catch the PR's own type-mismatch case. On this branch, running
`mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false
-DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest
-Dtest=none test`
still fails at `PaimonOptionTest.scala:331`: the insert reaches
`PaimonV2DataWriter` and aborts the Spark job with a `SparkException` caused by
`NegativeArraySizeException`, instead of failing here with the new
`PaimonSparkSessionExtensions` validation message. That means `dataSchema` is
not a reliable proxy for the physical query rows in this write path (or the
validation is happening at the wrong layer), so the original
fail-fast/corruption path remains unfixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
