harshmotw-db commented on code in PR #53120:
URL: https://github.com/apache/spark/pull/53120#discussion_r2547478465
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1593,6 +1593,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PARQUET_IGNORE_VARIANT_ANNOTATION =
+ buildConf("spark.sql.parquet.ignoreVariantAnnotation")
+ .doc("When true, ignore the variant logical type annotation and treat
the Parquet " +
+ "column in the same way as the underlying struct type")
+ .version("4.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Correct me if I'm wrong but I don't think logDebug would be helpful here if
we want to extract variant columns into a custom schema in a Spark DataFrame.
It is a good tool to debug issues in a Parquet file
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1593,6 +1593,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PARQUET_IGNORE_VARIANT_ANNOTATION =
+ buildConf("spark.sql.parquet.ignoreVariantAnnotation")
+ .doc("When true, ignore the variant logical type annotation and treat
the Parquet " +
+ "column in the same way as the underlying struct type")
+ .version("4.1.0")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Correct me if I'm wrong but I don't think logDebug would be helpful here if
we want to extract variant columns into a custom schema in a Spark DataFrame.
This config is a good tool to debug issues in a Parquet file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
