dongjoon-hyun opened a new pull request, #57881:
URL: https://github.com/apache/spark/pull/57881
### What changes were proposed in this pull request?
This PR aims to add `OpenHashMapBenchmark` to measure the performance of
`org.apache.spark.util.collection.OpenHashMap` against `java.util.HashMap`
in three scenarios
modeled on Spark's real usage:
- **Insert**: inserting 1M distinct String keys via `update`/`put`
- **Aggregate**: 5M `changeValue`/`merge` operations over 1M String keys
(the typical
aggregation pattern, e.g. `Mode`, `countByValue`)
- **Lookup**: looking up 1M String keys in random order
### Why are the changes needed?
The class documentation of `OpenHashMap` claims it is "about 5X faster than
java.util.HashMap".
This claim dates from 2013 (pre-JDK 8), and there has been no benchmark in
the repository to
verify it. On modern JDKs, `java.util.HashMap` has improved significantly,
and a local run on
JDK 21 shows the claim no longer holds
Note that `OpenHashMap` still uses about 2.2x less memory than
`java.util.HashMap` for
`String -> Long` entries thanks to its specialized primitive value storage.
This benchmark
provides a consistent way to track the trade-off across JDK versions and
future improvements.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Manually ran the benchmark:
```
build/sbt "core/Test/runMain
org.apache.spark.util.collection.OpenHashMapBenchmark"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
