JingsongLi commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3527492282
##########
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/PaimonOptionTest.scala:
##########
@@ -305,3 +305,41 @@ class PaimonConfigCheckTest extends SparkFunSuite {
}
}
}
+
+class PaimonV2WriteSchemaValidationTest extends SparkFunSuite with
SparkVersionSupport {
+
+ test("Paimon V2 write: fail fast on incompatible query schema without Paimon
extensions") {
+ assume(gteqSpark3_4)
+ val spark = SparkSession
+ .builder()
+ .master("local[2]")
+ .config("spark.sql.catalog.paimon",
"org.apache.paimon.spark.SparkCatalog")
+ .config("spark.sql.catalog.paimon.warehouse",
Utils.createTempDir.getCanonicalPath)
+ .config("spark.paimon.requiredSparkConfsCheck.enabled", "false")
+ .config("spark.paimon.write.use-v2-write", "true")
+ .getOrCreate()
+ try {
+ spark.sql("USE paimon")
+ spark.sql("CREATE TABLE T (a STRING, b STRING, pt STRING) PARTITIONED BY
(pt)")
+
+ // A query schema that matches the table schema still works without the
extensions.
+ spark.sql("INSERT INTO T VALUES ('x', 'y', '20260601')")
+ assert(spark.sql("SELECT * FROM T").collect() === Array(Row("x", "y",
"20260601")))
+
+ // Type mismatch: without PaimonAnalysis no cast is added, the write
must fail fast
+ // instead of corrupting data or throwing NegativeArraySizeException in
the writer.
+ val typeMismatch = intercept[RuntimeException] {
Review Comment:
Running the added suite locally with `mvn -pl paimon-spark/paimon-spark-ut
-am -Pfast-build -DfailIfNoTests=false
-DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest
-Dtest=none test` fails at this assertion: this insert still reaches the writer
and throws a `SparkException` caused by `NegativeArraySizeException`, rather
than the new fail-fast error mentioning `PaimonSparkSessionExtensions`. That
means the validation is not protecting the dynamic-partition overwrite path
covered by this test. Could you adjust the guard/test so this statement is
actually rejected before execution reaches the data writer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
