harshmotw-db commented on code in PR #53120:
URL: https://github.com/apache/spark/pull/53120#discussion_r2547336965
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1593,6 +1593,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PARQUET_IGNORE_VARIANT_ANNOTATION =
+ buildConf("spark.sql.parquet.ignoreVariantAnnotation")
+ .doc("When true, ignore the variant logical type annotation and treat
the Parquet " +
+ "column in the same way as the underlying struct type")
+ .version("4.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
It's mainly for debugging purposes if we need to extract the raw variant
bytes by specifying the schema as say `struct<value: Binary, metadata: Binary>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
