dongjoon-hyun edited a comment on issue #24286: [SPARK-27351][SQL] Wrong outputRows estimation after AggregateEstimation with only null value column URL: https://github.com/apache/spark/pull/24286#issuecomment-495351431 @rxin , @gatorsmile , @cloud-fan , @HyukjinKwon . Sorry, but I'm wondering if that is a correct reason for revert. - First, this PR doesn't affect `SQL COUNT DISTINCT` result and `DataSet.stats` result at all. - Second, this PR doesn't affect single column statistics like the following in Spark tables. ``` spark.sql.statistics.colStats.a.distinctCount=1, spark.sql.statistics.colStats.a.nullCount=1, spark.sql.statistics.colStats.b.distinctCount=0, spark.sql.statistics.colStats.b.nullCount=2, ``` - Although this is might be exposed to users, this is a meaningful fix for internal `AggregateEstimation` in Apache Spark CBO. Do we really need to revert this to prevent mismatch with `Pandas` or other user facing SQL output?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
