dongjoon-hyun commented on code in PR #53120:
URL: https://github.com/apache/spark/pull/53120#discussion_r2548105162
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1593,6 +1593,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PARQUET_IGNORE_VARIANT_ANNOTATION =
+ buildConf("spark.sql.parquet.ignoreVariantAnnotation")
+ .doc("When true, ignore the variant logical type annotation and treat
the Parquet " +
+ "column in the same way as the underlying struct type")
+ .version("4.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
May I ask why you think it that way? You told me that `It's mainly for
debugging purposes`, right?
> Correct me if I'm wrong but I don't think logDebug would be helpful here
if we want to extract variant columns into a custom schema in a Spark
DataFrame. This config is a good tool to debug issues in a Parquet file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
