uros-b commented on code in PR #58496:
URL: https://github.com/apache/spark/pull/58496#discussion_r3955944208
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala:
##########
@@ -2009,6 +2009,37 @@ class ParquetIOSuite extends ParquetTest with
SharedSparkSession {
}
}
+ test("SPARK-59251: Parquet readers reject incompatible primitive type
conversions consistently") {
+ val cases = Seq(
+ ("required int32 c (DATE);", DecimalType(10, 0),
+ (record: SimpleGroup) => record.add(0, 1)),
+ ("required fixed_len_byte_array(4) c;", StringType,
+ (record: SimpleGroup) =>
+ record.add(0, Binary.fromConstantByteArray(Array[Byte](1, 2, 3,
4)))))
+
+ cases.foreach { case (column, readType, writeValue) =>
+ val parquetSchema = MessageTypeParser.parseMessageType(
+ s"message root {\n $column\n}")
+ val readSchema = new StructType().add("c", readType)
+
+ withTempDir { dir =>
+ val path = new Path(s"${dir.getCanonicalPath}/incompatible.parquet")
+ val writer = createParquetWriter(parquetSchema, path)
+ val record = new SimpleGroup(parquetSchema)
+ writeValue(record)
+ writer.write(record)
+ writer.close()
+
+ withAllParquetReaders {
+ val error = intercept[SparkException] {
+ spark.read.schema(readSchema).parquet(path.toString).collect()
+ }
+ assert(error.getCondition ===
"FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH")
Review Comment:
Please use checkErrorMatchPVals (as in SPARK-35640 and the vectorized
schema-mismatch test) and cover an extra silent-corruption case such as INT64
TIMESTAMP → DECIMAL. A condition-only assert will pass even when column is c vs
[c].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
