kerwin-zk commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3523356787
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2WriteBuilder.scala:
##########
@@ -55,4 +61,63 @@ class PaimonV2WriteBuilder(table: FileStoreTable,
dataSchema: StructType, option
}
override def partitionRowType(): RowType =
table.schema().logicalPartitionType()
+
+ /**
+ * Fail fast when the query schema is binary-incompatible with the table
schema.
+ *
+ * Paimon tables declare `ACCEPT_ANY_SCHEMA`, so Spark skips its own output
resolution and the
+ * query is aligned to the table schema by `PaimonAnalysis` instead, which
is only injected when
+ * `PaimonSparkSessionExtensions` is configured. If a write is planned
without it, a
+ * type-mismatched query would reach the writer as-is and be interpreted
with the table's row
+ * type, silently corrupting data or failing with errors like
`NegativeArraySizeException` deep
+ * inside the format writer.
+ */
+ private def validateDataSchema(): Unit = {
+ // Row-level operations write the table's own rows back; schema evolution
intentionally
+ // diverges from the current table schema and is validated when merging.
+ if (copyOnWriteScan.nonEmpty ||
SchemaEvolutionHelper.mergeSchemaEnabled(options)) {
+ return
+ }
+
+ val expectedFields =
SparkTypeUtils.fromPaimonRowType(table.rowType()).fields
+ val actualFields =
SparkSystemColumns.filterSparkSystemColumns(dataSchema).fields
Review Comment:
Thanks for checking! I believe that run picked up a stale
`paimon-spark-common_2.12-1.5-SNAPSHOT` from the local repository instead of
the one from this branch, so the PR's change was never on the classpath.
`-Pfast-build` explicitly activates a profile, which disables the
`activeByDefault` activation of the `spark3` profile (standard Maven behavior).
Without `spark3`, the `paimon-spark-common` modules drop out of the reactor —
the reactor summary of that exact command only contains:
```
[INFO] Paimon : Spark
[pom]
[INFO] Paimon : Spark : UT : 2.12
[jar]
```
so `-am` never compiles this PR's change to `PaimonV2WriteBuilder`, and the
test runs against whatever `1.5-SNAPSHOT` is installed in `~/.m2`. If that
snapshot predates this PR, the test fails with exactly the signature you saw —
I actually produced the same message on purpose while developing (ran the new
test against a build with the fix stashed to verify it was red first):
```
Expected exception java.lang.RuntimeException to be thrown,
but org.apache.spark.SparkException was thrown (PaimonOptionTest.scala:331)
```
Adding the spark3 profile brings the common modules back into the reactor:
```
mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build,spark3
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest
-Dtest=none clean test
```
and the suite passes (your exact command also passes once the branch's
`paimon-spark-common` is installed to the local repo; the PR CI, which builds
from scratch, is green on both 2.12 and 2.13 as well):
```
- Paimon V2 write: fail fast on incompatible query schema without Paimon
extensions
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
```
On the layering question: for this insert path `LogicalWriteInfo.schema()`
is the analyzed query's output schema, which is exactly the row layout the
writer receives, so validating it against the table row type in `build()`
catches the mismatch on the driver before any task runs. Row-level operations
(`copyOnWriteScan`) and `write.merge-schema` writes are exempted since their
schemas intentionally diverge and are validated elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
