HyukjinKwon commented on code in PR #59080:
URL: https://github.com/apache/spark/pull/59080#discussion_r4172603597
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala:
##########
@@ -2018,6 +2018,54 @@ class ParquetIOSuite extends ParquetTest with
SharedSparkSession {
}
}
+ test("SPARK-59251: Parquet readers reject incompatible primitive type
conversions consistently") {
+ val cases: Seq[(String, DataType, SimpleGroup => Unit, String, String)] =
Seq(
+ ("required int32 c (DATE);", DecimalType(10, 0),
+ (record: SimpleGroup) => record.add(0, 1), "INT32", "decimal(10,0)"),
+ ("required fixed_len_byte_array(4) c;", StringType,
+ (record: SimpleGroup) =>
+ record.add(0, Binary.fromConstantByteArray(Array[Byte](1, 2, 3, 4))),
+ "FIXED_LEN_BYTE_ARRAY", "string"),
+ ("required int96 c;", StringType,
+ (record: SimpleGroup) =>
+ record.add(0, Binary.fromConstantByteArray(new Array[Byte](12))),
+ "INT96", "string"),
+ ("required int64 c (TIMESTAMP(MICROS,true));", DecimalType(20, 0),
+ (record: SimpleGroup) => record.add(0, 123456789L),
+ "INT64", "decimal(20,0)")
+ )
+
+ cases.foreach { case (column, readType, writeValue, actualType,
expectedType) =>
+ val parquetSchema = MessageTypeParser.parseMessageType(
+ s"message root {\n $column\n}")
+ val readSchema = new StructType().add("c", readType)
+
+ withTempDir { dir =>
+ val path = new Path(s"${dir.getCanonicalPath}/incompatible.parquet")
+ val writer = createParquetWriter(parquetSchema, path)
+ val record = new SimpleGroup(parquetSchema)
+ writeValue(record)
+ writer.write(record)
+ writer.close()
+
+ withAllParquetReaders {
+ checkErrorMatchPVals(
+ exception = intercept[SparkException] {
+ spark.read.schema(readSchema).parquet(path.toString).collect()
+ },
+ condition = "FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH",
+ parameters = Map(
+ "path" -> ".*",
+ "column" -> "\\[c\\]",
Review Comment:
**Nit (P3):** The fieldPath threading added across the converters is there
to report the full nested path in the PARQUET_COLUMN_DATA_TYPE_MISMATCH error,
but the added cases only exercise top-level incompatible conversions, so the
nested columnPath construction is never asserted. Since the nested path is
built from parquet field getName segments and (as the PR notes) may not
byte-for-byte match the vectorized reader's path, a divergence or regression
there would ship silently. Add a case that reads a Parquet file with an
incompatible conversion inside an array element, map key/value, or struct field
and asserts the error's `column` parameter is the full nested path.
**Verification:**
- **Regression:** A nested incompatible-conversion read fails with
PARQUET_COLUMN_DATA_TYPE_MISMATCH whose column parameter is the full nested
path, and the test fails if that path is wrong or diverges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
