Github user concretevitamin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1238#discussion_r15024063
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
---
@@ -26,6 +26,28 @@ import org.apache.spark.sql.catalyst.trees
abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
self: Product =>
+ // TODO: handle overflow?
+ /**
+ * Estimates of various statistics. The default estimation logic simply
sums up the corresponding
+ * statistic produced by the children. To override this behavior,
override `statistics` and
+ * assign it a overriden version of `Statistics`.
+ */
+ case class Statistics(
+ /**
+ * Number of output tuples. For leaf operators this defaults to 1,
otherwise it is set to the
+ * product of children's `numTuples`.
+ */
+ numTuples: Long = childrenStats.map(_.numTuples).product,
+
+ /**
+ * Physical size in bytes. For leaf operators this defaults to 1,
otherwise it is set to the
+ * product of children's `sizeInBytes`.
+ */
+ sizeInBytes: Long = childrenStats.map(_.sizeInBytes).product
+ )
+ lazy val statistics: Statistics = new Statistics
+ lazy val childrenStats = children.map(_.statistics)
--- End diff --
Thanks for the note -- I wasn't well aware of the memory/time overhead of
lazy vals. Since we have one statistic at the moment, `childrenStats` is used
only once for a logical node, so I went ahead and removed it.
However, memoizing `statistics` itself probably would save some planning
time, as it will get called multiple times during planning (e.g. different
cases in the HashJoin pattern match). If we were to use a `def` it is going to
be evaluated in the quadratic order.
Using `val` only will fire off the computation of the stats for all logical
nodes right away. I think this might be less desirable to only evaluating it
for everything under a Join when needed by a Strategy. Also, we'd have very few
operators in a typical query anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
