david-mollitor-db opened a new pull request, #58728:
URL: https://github.com/apache/spark/pull/58728
### What changes were proposed in this pull request?
`UTF8String.hashCode()` recomputed a Murmur3 hash over **all** of the
string's bytes on every call:
```java
public int hashCode() {
return Murmur3_x86_32.hashUnsafeBytes(base, offset, numBytes, 42);
}
```
This PR caches the result in a non-volatile `int`, using the modern
`java.lang.String` technique so a
legitimately zero hash is remembered (via a `hashIsZero` flag) rather than
recomputed on every call:
```java
private int hash;
private boolean hashIsZero;
public int hashCode() {
int h = hash;
if (h == 0 && !hashIsZero) {
h = Murmur3_x86_32.hashUnsafeBytes(base, offset, numBytes, 42);
if (h == 0) {
hashIsZero = true;
} else {
hash = h;
}
}
return h;
}
```
The cache is reset alongside `base`/`offset`/`numBytes` in `readExternal`
and `read` (Kryo) — the
only places a `UTF8String` is re-pointed to different memory.
### Why are the changes needed?
For a `UTF8String` hashed more than once — e.g. a key in a Scala
`Map`/`Set`, or the repeated
`contains`/`apply`/`update` probes against an aggregation buffer —
recomputing an O(length) Murmur3
scan each time is wasteful. Caching the hash (as `java.lang.String` does)
removes the repeated scans.
The change is safe for the same reason the class's existing lazily-computed
caches (`numChars`,
`isFullAscii`, `isValid`, `numBytesValid`) are safe: a `UTF8String` object's
value is stable for its
lifetime — callers `copy()` before retaining a value across a Tungsten
buffer reuse (see
`InternalRow.copyValue`). A cached hash relies on the exact same contract,
and is reset when the
fields are re-pointed during deserialization.
Memory: one `int` plus one `boolean`, which should largely be absorbed by
the object's existing
alignment padding.
### Does this PR introduce _any_ user-facing change?
No. The cached value equals the previously computed hash for every input.
### How was this patch tested?
Added a `UTF8StringSuite` test covering hash stability across repeated
calls, equals/hashCode
consistency for equal strings built different ways, and a Java-serialization
round-trip (which
exercises the `readExternal` reset). Existing `UTF8StringSuite` passes;
checkstyle is clean.
Note: the benefit applies to `UTF8String`s hashed multiple times; ephemeral
per-row views hashed once
see no change, and `BytesToBytesMap` (hash aggregate/join) hashes raw bytes
directly rather than
through `UTF8String.hashCode`, so it is unaffected.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
