wangyum opened a new pull request, #36628:
URL: https://github.com/apache/spark/pull/36628
### What changes were proposed in this pull request?
Switch decimal type divide from
```
toJavaBigDecimal.divide(that.toJavaBigDecimal, MATH_CONTEXT)
```
to
```
toJavaBigDecimal.divide(that.toJavaBigDecimal, DecimalType.MAX_SCALE,
MATH_CONTEXT.getRoundingMode)
```
### Why are the changes needed?
Improve divide performance for decimal type.
Benchmark code:
```scala
import org.apache.spark.benchmark.Benchmark
val valuesPerIteration = 2880404L
val dir = "/tmp/spark/benchmark"
spark.range(valuesPerIteration).selectExpr("CAST(id AS DECIMAL(9, 2)) AS
d").write.mode("Overwrite").parquet(dir)
val benchmark = new Benchmark("Benchmark decimal", valuesPerIteration,
minNumIters = 5)
benchmark.addCase("d * 2 > 0") { _ =>
spark.read.parquet(dir).where("d * 2 >
0").write.format("noop").mode("Overwrite").save()
}
benchmark.addCase("d / 2 > 0") { _ =>
spark.read.parquet(dir).where("d / 2 >
0").write.format("noop").mode("Overwrite").save()
}
benchmark.run()
```
Before this PR:
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark decimal: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
d * 2 > 0 480 585
141 6.0 166.7 1.0X
d / 2 > 0 4689 4920
243 0.6 1627.9 0.1X
```
After this PR:
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark decimal: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
d * 2 > 0 529 580
35 5.4 183.6 1.0X
d / 2 > 0 811 916
80 3.6 281.4 0.7X
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
