leerho opened a new issue, #191: URL: https://github.com/apache/datasketches-go/issues/191
@proost, `GenerateHashKeyFromStrings` in `tuple/arrayofstrings_sketch.go` hashes the UTF-8 bytes of the key strings. Java's `tuple/Util.stringArrHash` hashes the joined key as UTF-16 code units (`XxHash.hashCharArr`, 2 bytes per `char`, little-endian), with the same seed and `,` separator. As a result, Go and Java AoS sketches built from the same keys share no hashes, and cross-language unions and intersections are wrong. This was confirmed with the `aos_*` snapshots in datasketches-tck: Go's snapshots match C++'s, and neither matches Java's. Java's AoS has been released for years and is the reference. The fix is to convert each key from UTF-8 to UTF-16 code units, with surrogate pairs above U+FFFF, and hash them as little-endian bytes with `,` as `2c 00`. In Go, `utf16.Encode([]rune(s))` gives the code units. Please also add a test that checks hashes against the Java snapshots, including `aos_unicode`. Note that the AoS sketch was released in v0.2.0, so this change invalidates AoS sketches saved with v0.2.0. Please call it out in the release notes. The same issue in C++ is apache/datasketches-cpp#533, which we are fixing for the C++ 5.3.0 release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
