proost opened a new pull request, #197:
URL: https://github.com/apache/datasketches-go/pull/197
## Summary
Follow-up to #196 for #191.
#196 made `GenerateHashKeyFromStrings` hash keys as UTF-16LE, as Java does,
but it decoded each string by ranging over it. That gives one U+FFFD per
invalid byte. Java only hashes invalid UTF-8 after its UTF-8 decoder (`new
String(bytes, UTF_8)`) has turned the bytes into a `String`, and that decoder
replaces them differently in two cases:
| Input | Java | #196 |
|---|---|---|
| `E2 82 41` (cut-off 3-byte character) | `FFFD 0041` | `FFFD FFFD 0041` |
| `F0 9F 94 41` (cut-off 4-byte character) | `FFFD 0041` | `FFFD FFFD FFFD
0041` |
| `ED A0 80` (encoded surrogate) | `FFFD` | `FFFD FFFD FFFD` |
This PR decodes the bytes with the same rules as the JDK's
`String.decodeUTF8_UTF16`: the prefix lengths of `malformed3` and `malformed4`,
the surrogate check, and the end-of-input handling. Valid UTF-8 hashes exactly
as before.
## Verification
- **Differential test against real JDKs:** every 1–4 byte sequence over 27
boundary bytes (one or two from each byte class a UTF-8 decoder distinguishes),
plus 200,000 random 5–10 byte sequences, for 751,880 inputs in total.
- JDK 8, 11, 17, 21, 25 and 26 all produce identical output.
- `new String(b, UTF_8)` and `CharsetDecoder` with `REPLACE` agree on
every input.
- This PR has 0 mismatches. #196 has 167,194.
- **`TestGenerateHashKeyFromStrings_JavaDecoding`:** 20 cases with expected
values taken from Java, including inputs cut off at the end.
- `make test` (with the TCK snapshots) and `make lint` pass.
## Compatibility
#196 is not in any release, so this adds no break beyond the one v0.3.0
already has for AoS key hashing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
