viirya commented on a change in pull request #27344: [SPARK-30622][SQL]
commands should return dummy statistics
URL: https://github.com/apache/spark/pull/27344#discussion_r370462413
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
##########
@@ -27,4 +27,8 @@ import org.apache.spark.sql.catalyst.expressions.Attribute
trait Command extends LogicalPlan {
override def output: Seq[Attribute] = Seq.empty
override def children: Seq[LogicalPlan] = Seq.empty
+ // Commands are eagerly executed. They will be converted to LocalRelation
after the DataFrame
+ // is created. That said, the statistics of a command is useless. Here we
just return a dummy
+ // statistics to avoid unnecessary statistics calculation of command's
children.
+ override def stats: Statistics = Statistics(Long.MaxValue)
Review comment:
yea, +1 for the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
