leerho commented on issue #191:
URL:
https://github.com/apache/datasketches-go/issues/191#issuecomment-5940780932
Two additions to include in the fix:
1. **Hash input, precisely:** the UTF-16 code units of the joined key, as
**little-endian bytes with no byte-order mark** (UTF-16LE). This is what Java's
`XxHash.hashCharArr(s.toCharArray(), ...)` hashes. Note that a generic "UTF-16"
converter is not the same thing: Java's own `StandardCharsets.UTF_16`, for
example, writes a `FE FF` byte-order mark followed by big-endian bytes.
2. **Document the key-join behavior.** Key elements are joined with `","`
without escaping, so different keys that join to the same string hash the same
and are treated as one key, for example `{"a,b"}` and `{"a", "b"}`, or an empty
array and `{""}`. We can't change this without breaking compatibility with
Java, so all implementations must keep this behavior. Please note it in the API
documentation, along these lines:
> Key arrays are hashed by joining their elements with `","`. Keys that
join to the same string, for example `{"a,b"}` and `{"a", "b"}`, are treated as
the same key. If key elements may contain commas, escape or encode them before
updating the sketch.
We'll add the same note to the Java documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
