MaxGekk commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1766340208
##########
sql/core/benchmarks/DateTimeRebaseBenchmark-jdk21-results.txt:
##########
@@ -6,109 +6,109 @@ OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux
6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
Save DATE to parquet: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 11193 11193
0 8.9 111.9 1.0X
-before 1582, noop 7070 7070
0 14.1 70.7 1.6X
-after 1582, rebase EXCEPTION 19836 19836
0 5.0 198.4 0.6X
-after 1582, rebase LEGACY 19368 19368
0 5.2 193.7 0.6X
-after 1582, rebase CORRECTED 19627 19627
0 5.1 196.3 0.6X
-before 1582, rebase LEGACY 16301 16301
0 6.1 163.0 0.7X
-before 1582, rebase CORRECTED 15612 15612
0 6.4 156.1 0.7X
+after 1582, noop 11734 11734
0 8.5 117.3 1.0X
+before 1582, noop 7155 7155
0 14.0 71.6 1.6X
+after 1582, rebase EXCEPTION 20374 20374
0 4.9 203.7 0.6X
+after 1582, rebase LEGACY 19681 19681
0 5.1 196.8 0.6X
+after 1582, rebase CORRECTED 20160 20160
0 5.0 201.6 0.6X
+before 1582, rebase LEGACY 16245 16245
0 6.2 162.5 0.7X
+before 1582, rebase CORRECTED 16039 16039
0 6.2 160.4 0.7X
Review Comment:
Not related to this PR, but `Stdev` is all `0` confuses slightly.
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -285,12 +285,12 @@ object RebaseDateTime {
}
// Loads rebasing info from an JSON file. JSON records in the files should
conform to
- // `JsonRebaseRecord`. AnyRefMap is used here instead of Scala's immutable
map because
- // it is 2 times faster in DateTimeRebaseBenchmark.
- private[sql] def loadRebaseRecords(fileName: String): AnyRefMap[String,
RebaseInfo] = {
+ // `JsonRebaseRecord`. Mutable HashMap is used here instead of AnyRefMap due
to SPARK-49491.
+ private[sql] def loadRebaseRecords(fileName: String): HashMap[String,
RebaseInfo] = {
val file = SparkClassUtils.getSparkClassLoader.getResource(fileName)
val jsonRebaseRecords = mapper.readValue[Seq[JsonRebaseRecord]](file)
- val anyRefMap = new AnyRefMap[String, RebaseInfo]((3 *
jsonRebaseRecords.size) / 2)
+ val hashMap = new HashMap[String, RebaseInfo]
+ hashMap.sizeHint((3 * jsonRebaseRecords.size) / 2)
Review Comment:
Please, set just `jsonRebaseRecords.size`. `HashMap` has some internal
heuristic already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
