maropu commented on a change in pull request #29983:
URL: https://github.com/apache/spark/pull/29983#discussion_r502364642
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2775,6 +2775,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR =
+ buildConf("spark.sql.legacy.centralMomentAgg.enabled")
Review comment:
Could you update the migration guide, too?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2775,6 +2775,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR =
Review comment:
nit: `LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR` -> `LEGACY_CENTRAL_MOMENT_AGG`
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2775,6 +2775,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR =
+ buildConf("spark.sql.legacy.centralMomentAgg.enabled")
Review comment:
Looks we don't need the suffix `.enabled` for following the other legacy
configs.
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2775,6 +2775,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR =
+ buildConf("spark.sql.legacy.centralMomentAgg.enabled")
+ .internal()
+ .doc("When set to true, stddev_samp and var_samp will return Double.NaN,
" +
+ "if applied to a set with a single element. Otherwise, will return
0.0, " +
+ "which is aligned with TPCDS standard.")
Review comment:
I think we don't need to describe `which is aligned with TPCDS
standard.` here for user documents.
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2775,6 +2775,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR =
+ buildConf("spark.sql.legacy.centralMomentAgg.enabled")
Review comment:
Also, could you move this config close to the other legacy configs?
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
##########
@@ -456,25 +456,31 @@ class DataFrameAggregateSuite extends QueryTest
}
test("zero moments") {
Review comment:
How about organizing tests like this? (I think it'd better not to update
the existing tests as much as possible):
```
test("zero moments") {
withSQLConf(SQLConf.LEGACY_CENTRAL_MOMENT_AGG_BEHAVIOR.key -> "true") {
// Don't touch the existing tests
val input = Seq((1, 2)).toDF("a", "b")
checkAnswer(
input.agg(stddev($"a"), stddev_samp($"a"), stddev_pop($"a"),
variance($"a"),
var_samp($"a"), var_pop($"a"), skewness($"a"), kurtosis($"a")),
Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN, 0.0,
Double.NaN, Double.NaN))
checkAnswer(
input.agg(
expr("stddev(a)"),
expr("stddev_samp(a)"),
expr("stddev_pop(a)"),
expr("variance(a)"),
expr("var_samp(a)"),
expr("var_pop(a)"),
expr("skewness(a)"),
expr("kurtosis(a)")),
Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN, 0.0,
Double.NaN, Double.NaN))
}
}
test("SPARK-13860: xxxx") {
// Writes tests for the new behaviour
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
