sunchao commented on code in PR #57443:
URL: https://github.com/apache/spark/pull/57443#discussion_r3641558475
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -765,20 +765,31 @@ object SQLConf {
val RUNTIME_BLOOM_FILTER_ENABLED =
buildConf("spark.sql.optimizer.runtime.bloomFilter.enabled")
- .doc("When true and if one side of a shuffle join has a selective
predicate, we attempt " +
- "to insert a bloom filter in the other side to reduce the amount of
shuffle data.")
+ .doc("When true and if one side of a shuffle join has a selective
predicate, or is a " +
+ "fully materialized, repeatable cache with evidence that pruning is
beneficial, we " +
+ "attempt to insert a bloom filter in the other side to reduce shuffle
data.")
.version("3.3.0")
.booleanConf
.createWithDefault(true)
val RUNTIME_BLOOM_FILTER_CREATION_SIDE_THRESHOLD =
buildConf("spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold")
.doc("Size threshold of the bloom filter creation side plan. Estimated
size needs to be " +
- "under this value to try to inject bloom filter.")
+ "under this value to try to inject a bloom filter, unless the creation
side is fully " +
+ "materialized and has accurate statistics.")
.version("3.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10MB")
+ val RUNTIME_BLOOM_FILTER_MATERIALIZED_CREATION_SIDE_THRESHOLD =
Review Comment:
Good catch. I added
`.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)`. This threshold only
changes runtime-filter optimization; it does not affect the resolved body of a
view, UDF, or procedure, so `NOT_APPLICABLE` is the appropriate binding policy.
I also verified the change with `SparkConfigBindingPolicySuite`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -765,20 +765,31 @@ object SQLConf {
val RUNTIME_BLOOM_FILTER_ENABLED =
buildConf("spark.sql.optimizer.runtime.bloomFilter.enabled")
- .doc("When true and if one side of a shuffle join has a selective
predicate, we attempt " +
- "to insert a bloom filter in the other side to reduce the amount of
shuffle data.")
+ .doc("When true and if one side of a shuffle join has a selective
predicate, or is a " +
+ "fully materialized, repeatable cache with evidence that pruning is
beneficial, we " +
+ "attempt to insert a bloom filter in the other side to reduce shuffle
data.")
.version("3.3.0")
.booleanConf
.createWithDefault(true)
val RUNTIME_BLOOM_FILTER_CREATION_SIDE_THRESHOLD =
buildConf("spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold")
.doc("Size threshold of the bloom filter creation side plan. Estimated
size needs to be " +
- "under this value to try to inject bloom filter.")
+ "under this value to try to inject a bloom filter, unless the creation
side is fully " +
+ "materialized and has accurate statistics.")
.version("3.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10MB")
+ val RUNTIME_BLOOM_FILTER_MATERIALIZED_CREATION_SIDE_THRESHOLD =
+
buildConf("spark.sql.optimizer.runtime.bloomFilter.materializedCreationSideThreshold")
+ .doc("Size threshold of a fully materialized, repeatable bloom filter
creation side with " +
+ "accurate statistics. This replaces the general creation-side
threshold because scanning " +
+ "materialized output does not recompute its original plan.")
+ .version("5.0.0")
Review Comment:
Agreed. I changed this to `.version("4.3.0")`: this is an optimizer
improvement that can ship on the 4.x line, not a 5.0-only change, so `4.3.0` is
the correct first-release version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
