jubins opened a new pull request, #29946: URL: https://github.com/apache/flink/pull/29946
## What is the purpose of the change Fixes [FLINK-41503](https://issues.apache.org/jira/browse/FLINK-41503). `MurmurHashUtil` in `flink-table-runtime` has no unit tests. `SegmentsUtil.hash` / `hashByWords` and the generated hash code for `BINARY` / `VARBINARY` / `UUID` fields (`CodeGenUtils.hashCodeForType`) call it, and `IntHashSet`, `LongHashSet` and similar sets use its `fmix` methods. If a hash value changed by accident, keys would be assigned to different key groups and partitions, and restoring existing state would break. Nothing currently catches that. This PR adds tests only. `BinaryRowData` and `BinaryStringData` hash with a package-private copy of the same algorithm in `flink-table-common` (`org.apache.flink.table.data.binary.MurmurHashUtils`). The new tests also check that the runtime class gives the same result as `BinaryStringData#hashCode`, so the two copies cannot drift apart without a failing test. ## Brief change log - `flink-table-runtime/.../runtime/util/MurmurHashUtilTest.java` (new): six JUnit 5 / AssertJ tests for `MurmurHashUtil`. No production code changes. ## Verifying this change This change added tests and can be verified as follows: - `testHashBytesMatchesHashUnsafeBytesForAllTailLengths`: `hashBytes(MemorySegment, ...)` equals `hashUnsafeBytes(byte[], ...)` for every length from 0 to 17, which covers tails of 0 to 3 bytes. The input includes negative bytes. - `testHashBytesByWordsMatchesHashUnsafeBytesByWords`: `hashBytesByWords` equals `hashUnsafeBytesByWords` and `hashUnsafeBytes` for lengths 0, 4, 8, 12 and 16. - `testHashBytesWithOffsetInsideSegment`: hashing at a non-zero offset inside a larger segment or array gives the same result as hashing the bytes alone. - `testFmixKnownValues`: `fmix(int)` and `fmix(long)` return the reference MurmurHash3 fmix32 / fmix64 outputs (`0x514e28b7` and `0xb456bcfc34c2cb2c` for input 1). - `testPinnedHashValues`: pins the outputs that master produces for the empty input (142593372), for "a", "abcd" and "Hello, Flink!", for a 3-byte tail with sign-extended bytes, and for an 8-byte by-words input. A comment in the test explains why these values are pinned. - `testMatchesBinaryStringDataHashCode`: `MurmurHashUtil.hashBytes` equals `BinaryStringData.fromString(s).hashCode()`. To check that the tests catch real changes, I changed `MurmurHashUtil` locally and then restored it: - Treating the segment tail byte as unsigned (`segment.get(offset + i) & 0xff`) made `testHashBytesMatchesHashUnsafeBytesForAllTailLengths` and `testPinnedHashValues` fail. - Changing `C1` from `0xcc9e2d51` to `0xcc9e2d53` made `testPinnedHashValues` and `testMatchesBinaryStringDataHashCode` fail. The equivalence tests alone cannot catch this kind of change. `mvn -pl flink-table/flink-table-runtime test -Dtest=MurmurHashUtilTest`: 6 tests, 0 failures. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
