Terry Kim created SPARK-28217:
---------------------------------
Summary: Allow a custom statistics logical plan visitor to be
plugged in.
Key: SPARK-28217
URL: https://issues.apache.org/jira/browse/SPARK-28217
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Terry Kim
Spark currently has two built-in statistics plan visitor:
SizeInBytesOnlyStatsPlanVisitor and BasicStatsPlanVisitor. However, this is a
bit limited since there is no way to plug in a custom plan visitor - from which
a custom query optimizer can benefit from.
We can provide a Spark conf that the user can specify to override the built-in
plan visitor:
{code:scala}
// First create your custom stat plan visitor.
class MyStatsPlanVisitor extends LogicalPlanVisitor[Statistics] {
// Implement LogicalPlanVisitor[Statistics] trait
}
// Set the visitor via Spark conf.
spark.conf.set("spark.sql.catalyst.statsPlanVisitorClass", "MyStatsPlanVisitor")
// Now, stat() on a LogicalPlan object will use MyStatsPlanVisitor as a stat
plan visitor.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
