toderesa97 commented on issue #21764:
URL: https://github.com/apache/spark/pull/21764#issuecomment-616719975
Hello,
It is not very clear for me, how to exclude some rules. I have been digging
into the [testing file for
OptimizerRuleExclusionSuite.scala](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerRuleExclusionSuite.scala)
and not able to exclude some rules. As the jira report for issue
[SPARK-24802](https://issues.apache.org/jira/browse/SPARK-24802) says:
> Add a new config spark.sql.optimizer.excludedRules, with the value being a
list of rule names separated by comma
I tried this:
```
import org.apache.spark.sql.{SparkSession}
import org.apache.spark.sql.catalyst.optimizer._
object Main {
def getExcludeRules: Seq[String] = {
Seq(
PushPredicateThroughJoin.ruleName
)
}
def main(args: Array[String]): Unit = {
val spark: SparkSession = SparkSession
.builder
.config("spark.sql.optimizer.excludedRules", getExcludeRules)
.master("local[*]")
.getOrCreate
val sc = spark.sparkContext
// whatever
}
}
```
but it is not working since there is not such a method ``config`` that
receives a second parameter a ``Seq[String]``.
Thank you for any help you can provide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
