harshmotw-db commented on code in PR #53120:
URL: https://github.com/apache/spark/pull/53120#discussion_r2547527820
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1593,6 +1593,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PARQUET_IGNORE_VARIANT_ANNOTATION =
+ buildConf("spark.sql.parquet.ignoreVariantAnnotation")
+ .doc("When true, ignore the variant logical type annotation and treat
the Parquet " +
+ "column in the same way as the underlying struct type")
+ .version("4.1.0")
Review Comment:
This PR practically is a fix already. [This
PR](https://github.com/apache/spark/pull/53005) added a temporary workaround
for reading variant data mainly for testing purposes ([see this
line](https://github.com/harshmotw-db/spark/blob/e308d209b873a0d3a1e6a2f04038eec8f2b55036/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L376-L377)).
Essentially, the existing code behaves as if `ignoreVariantAnnotation =
false`. This PR just implements this code more formally so we actually do make
sure that the target type matches the actual parquet type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
