peng bo created SPARK-27351:
-------------------------------
Summary: Wrong outputRows estimation after AggregateEstimation
with only null value column
Key: SPARK-27351
URL: https://issues.apache.org/jira/browse/SPARK-27351
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.1
Reporter: peng bo
The upper bound of group-by columns row number is to multiply distinct counts
of group-by columns. However, column with only null value will cause the output
row number to be 0 which is incorrect.
Ex:
col1 (distinct: 2, rowCount 2)
col2 (distinct: 0, rowCount 2)
group by col1, col2
Actual: output rows: 0
Expected: output rows: 2
{code:java}
var outputRows: BigInt = agg.groupingExpressions.foldLeft(BigInt(1))(
(res, expr) => res *
childStats.attributeStats(expr.asInstanceOf[Attribute]).distinctCount)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
