srowen commented on a change in pull request #26029: [SPARK-29336][SQL] Fix the
implementation of QuantileSummaries.merge (guarantee that the relativeError
will be respected)
URL: https://github.com/apache/spark/pull/26029#discussion_r331750977
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/ApproximatePercentileQuerySuite.scala
##########
@@ -124,20 +124,24 @@ class ApproximatePercentileQuerySuite extends QueryTest
with SharedSparkSession
test("percentile_approx, with different accuracies") {
withTempView(table) {
- (1 to 1000).toDF("col").createOrReplaceTempView(table)
+ val tableCount = 1000
+ (1 to tableCount).toDF("col").createOrReplaceTempView(table)
// With different accuracies
- val expectedPercentile = 250D
val accuracies = Array(1, 10, 100, 1000, 10000)
- val errors = accuracies.map { accuracy =>
- val df = spark.sql(s"SELECT percentile_approx(col, 0.25, $accuracy)
FROM $table")
- val approximatePercentile = df.collect().head.getInt(0)
- val error = Math.abs(approximatePercentile - expectedPercentile)
- error
+ val expectedPercentiles = Array(100D, 200D, 250D, 314D, 777D)
Review comment:
Minor nit: I'd write 100.0 instead of 100D etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
