This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new c529597537a3 [SPARK-58682][DOC] Update outdated
`OpenHashMap`/`OpenHashSet` performance claims
c529597537a3 is described below
commit c529597537a3e83efa6a2539fca9ef6d9a884789
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Aug 9 20:53:08 2026 -0700
[SPARK-58682][DOC] Update outdated `OpenHashMap`/`OpenHashSet` performance
claims
### What changes were proposed in this pull request?
This PR aims to update the outdated performance claims in the class
documentation of
`OpenHashMap` and `OpenHashSet`, based on the results of
`OpenHashMapBenchmark` added
by SPARK-58678.
- `OpenHashMap`: replaced the claim "about 5X faster than
java.util.HashMap" with an
accurate description: it uses much less space and is competitive for
aggregation
workloads (`changeValue`), while `java.util.HashMap` is faster for pure
insertions
and lookups on modern JDKs.
- `OpenHashSet`: replaced the unqualified claim "much faster than Java's
standard
HashSet" with the verifiable facts: much less memory overhead, and the
specialized
versions avoid boxing of primitive keys.
### Why are the changes needed?
The "about 5X faster" claim dates from 2013 (pre-JDK 8).
`OpenHashMapBenchmark`
(SPARK-58678) shows that on modern JDKs the claim no longer holds:
- **Insert**: `java.util.HashMap` is 4.6X, 5.5X, 6.5X faster on Java 17,
21, and 25,
respectively.
- **Lookup**: `java.util.HashMap` is about 2X faster.
- **Aggregate** (`changeValue`/`merge`): `OpenHashMap` is slightly faster.
Since the benchmark's insert/lookup paths with object keys effectively
measure
`OpenHashSet` as well, its unqualified "much faster" claim is also updated.
The
memory advantage (about 2.2x less for `String -> Long` entries) remains
true and is
kept.
### Does this PR introduce _any_ user-facing change?
No. This is a comment-only change.
### How was this patch tested?
Pass the CIs. This is a comment-only change with no behavior change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57885 from dongjoon-hyun/SPARK-58682.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 5adcd21e6073236b57ab2871d52df0a0e08c41d5)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../main/scala/org/apache/spark/util/collection/OpenHashMap.scala | 5 +++--
.../main/scala/org/apache/spark/util/collection/OpenHashSet.scala | 7 ++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
index e421a1f4746e..3c7f083381c3 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
@@ -21,8 +21,9 @@ import scala.reflect.ClassTag
/**
* A fast hash map implementation for nullable keys. This hash map supports
insertions and updates,
- * but not deletions. This map is about 5X faster than java.util.HashMap,
while using much less
- * space overhead.
+ * but not deletions. This map uses much less space than java.util.HashMap and
is competitive with
+ * it for aggregation workloads (`changeValue`), while java.util.HashMap is
faster for pure
+ * insertions and lookups on modern JDKs. See `OpenHashMapBenchmark` for
details.
*
* Under the hood, it uses our OpenHashSet implementation.
*
diff --git
a/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
b/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
index 3d1eb5788c70..c56ae0224740 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
@@ -28,9 +28,10 @@ import org.apache.spark.annotation.Private
* removed.
*
* The underlying implementation uses Scala compiler's specialization to
generate optimized
- * storage for four primitive types (Long, Int, Double, and Float). It is much
faster than Java's
- * standard HashSet while incurring much less memory overhead. This can serve
as building blocks
- * for higher level data structures such as an optimized HashMap.
+ * storage for four primitive types (Long, Int, Double, and Float). It incurs
much less memory
+ * overhead than Java's standard HashSet, and the specialized versions avoid
boxing of primitive
+ * keys. This can serve as building blocks for higher level data structures
such as an optimized
+ * HashMap.
*
* This OpenHashSet is designed to serve as building blocks for higher level
data structures
* such as an optimized hash map. Compared with standard hash set
implementations, this class
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
