jubins opened a new pull request, #29946:
URL: https://github.com/apache/flink/pull/29946

   ## What is the purpose of the change
   
   Fixes [FLINK-41503](https://issues.apache.org/jira/browse/FLINK-41503).
   
   `MurmurHashUtil` in `flink-table-runtime` has no unit tests. 
`SegmentsUtil.hash` / `hashByWords` and the generated hash code for `BINARY` / 
`VARBINARY` / `UUID` fields (`CodeGenUtils.hashCodeForType`) call it, and 
`IntHashSet`, `LongHashSet` and similar sets use its `fmix` methods. If a hash 
value changed by accident, keys would be assigned to different key groups and 
partitions, and restoring existing state would break. Nothing currently catches 
that. This PR adds tests only.
   
   `BinaryRowData` and `BinaryStringData` hash with a package-private copy of 
the same algorithm in `flink-table-common` 
(`org.apache.flink.table.data.binary.MurmurHashUtils`). The new tests also 
check that the runtime class gives the same result as 
`BinaryStringData#hashCode`, so the two copies cannot drift apart without a 
failing test.
   
   ## Brief change log
   
     - `flink-table-runtime/.../runtime/util/MurmurHashUtilTest.java` (new): 
six JUnit 5 / AssertJ tests for `MurmurHashUtil`. No production code changes.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - `testHashBytesMatchesHashUnsafeBytesForAllTailLengths`: 
`hashBytes(MemorySegment, ...)` equals `hashUnsafeBytes(byte[], ...)` for every 
length from 0 to 17, which covers tails of 0 to 3 bytes. The input includes 
negative bytes.
     - `testHashBytesByWordsMatchesHashUnsafeBytesByWords`: `hashBytesByWords` 
equals `hashUnsafeBytesByWords` and `hashUnsafeBytes` for lengths 0, 4, 8, 12 
and 16.
     - `testHashBytesWithOffsetInsideSegment`: hashing at a non-zero offset 
inside a larger segment or array gives the same result as hashing the bytes 
alone.
     - `testFmixKnownValues`: `fmix(int)` and `fmix(long)` return the reference 
MurmurHash3 fmix32 / fmix64 outputs (`0x514e28b7` and `0xb456bcfc34c2cb2c` for 
input 1).
     - `testPinnedHashValues`: pins the outputs that master produces for the 
empty input (142593372), for "a", "abcd" and "Hello, Flink!", for a 3-byte tail 
with sign-extended bytes, and for an 8-byte by-words input. A comment in the 
test explains why these values are pinned.
     - `testMatchesBinaryStringDataHashCode`: `MurmurHashUtil.hashBytes` equals 
`BinaryStringData.fromString(s).hashCode()`.
   
   To check that the tests catch real changes, I changed `MurmurHashUtil` 
locally and then restored it:
     - Treating the segment tail byte as unsigned (`segment.get(offset + i) & 
0xff`) made `testHashBytesMatchesHashUnsafeBytesForAllTailLengths` and 
`testPinnedHashValues` fail.
     - Changing `C1` from `0xcc9e2d51` to `0xcc9e2d53` made 
`testPinnedHashValues` and `testMatchesBinaryStringDataHashCode` fail. The 
equivalence tests alone cannot catch this kind of change.
   
   `mvn -pl flink-table/flink-table-runtime test -Dtest=MurmurHashUtilTest`: 6 
tests, 0 failures.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to