sunchao commented on a change in pull request #34308:
URL: https://github.com/apache/spark/pull/34308#discussion_r744953895
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
##########
@@ -1084,6 +1087,32 @@ class ParquetIOSuite extends QueryTest with ParquetTest
with SharedSparkSession
}
}
}
+
+ test("SPARK-37035: Improve error message when use parquet vectorized
reader") {
Review comment:
I spent sometime looking for a e2e test but couldn't figure out one 🤷 .
The errors with a corrupted Parquet file would either be caught when
initializing the Parquet dictionary (for instance, when trying to initialize a
Parquet INT64 dictionary with INT32 type), or in Spark when checking Spark
schema versus Parquet schema.
It's be interesting to know how you encountered this error originally
@AngersZhuuuu . Which Spark version you were using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
