viirya commented on code in PR #58050:
URL: https://github.com/apache/spark/pull/58050#discussion_r3799325391
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7078,6 +7037,22 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val VARIANT_SHREDDED_PREDICATE_PUSHDOWN_ENABLED =
+ buildConf("spark.sql.variant.shreddedPredicatePushdown.enabled")
+ .internal()
+ .doc("When true, comparison predicates on shredded Variant fields
produced by " +
+ "PushVariantIntoScan (e.g. variant_get(v, '$.a', 'bigint') > 999) are
pushed to Parquet " +
+ "as the predicate on the physical shredded typed_value leaf column
OR-ed with an " +
+ "IS NOT NULL check on every untyped residual value column along the
path, so that a row " +
+ "group is skipped only when the leaf cannot match and every residual
is entirely null " +
+ "(i.e. the whole path is provably in the typed leaf). This enables
row-group skipping " +
+ "for shredded Variant columns while never dropping rows that fall back
to an untyped " +
+ "residual. Has no effect unless the Parquet column is shredded and " +
+ "spark.sql.variant.pushVariantIntoScan is also true.")
+ .version("4.3.0")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
I will change the default value to false before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
