leerho opened a new issue, #16:
URL: https://github.com/apache/datasketches-cuda/issues/16
datasketches-cpp 5.3.0 is released. It changes the meaning of HLL flag bit
0x20, which datasketches-cuda currently sets on every image it writes. This
issue describes the change and what datasketches-cuda needs to do.
### What changed in C++ 5.3.0 (apache/datasketches-cpp#521)
Up to 5.2.0, C++ wrote bit 0x20 of the flags byte as `FULL_SIZE_FLAG_MASK`
("this sketch was constructed with `start_full_size`"). datasketches-java uses
the same bit as `REBUILD_CURMIN_NUM_KXQ_MASK`, which tells its union to rebuild
curMin, numAtCurMin and KxQ from the registers. The two meanings collided
across languages.
In 5.3.0 the bit is reserved: C++ neither writes nor reads it.
`hll_constants::FULL_SIZE_FLAG_MASK` remains only as a deprecated alias of
`RESERVED_FLAG_MASK_32`, so code that uses it still compiles.
Full size is now purely an in-memory construction choice. The
`hll_sketch(lg_k, type, start_full_size)` constructor is unchanged, and
`reset()` became `reset(bool full_size = false)`, because the choice is no
longer remembered across a reset or a serialization round trip. On read, the
mode byte alone (for example `0x0A`, HLL_8 in HLL mode) tells the reader to
build the full HLL array.
### What datasketches-cuda needs to do
1. Bump `GIT_TAG` from 5.2.0 to 5.3.0 in
`cmake/thirdparty/get_datasketches_cpp.cmake`.
2. Stop writing bit 0x20. `sketch_impl.hpp:143` sets `full_size_flag = true`
and `preamble.hpp:84` writes it as `FULL_SIZE_FLAG_MASK`. Leave the bit clear.
3. Ignore bit 0x20 on read (`preamble.hpp:138`). Don't reject images that
have it set, because C++ 5.2.0 and earlier wrote it for sketches constructed
with `start_full_size`.
4. Update the tests that expect the bit: `preamble_test.cpp` sets
`full_size_flag = true` and compares a round trip byte for byte with a C++
image (around line 96). With C++ 5.3.0, the C++ image has the bit clear.
5. If datasketches-cuda adds a `reset()`, mirror the C++ signature
`reset(bool full_size = false)`. The constructor can call it to start full size.
After this, datasketches-cuda images are byte-identical to C++ 5.3.0 images
for the same registers, and Java never sees a spurious rebuild flag.
### Also worth checking with 5.3.0
- apache/datasketches-cpp#522 pinned the HLL estimator constants and
finished the deferred KxQ rebuild. If any datasketches-cuda tests compare
estimates with C++ to the last bit, re-run them against 5.3.0.
- C++ 5.3.0 fixes five deserialization CVEs
(https://datasketches.apache.org/docs/Community/Security.html).
datasketches-cuda uses its own HLL deserializer, so it is not affected, but its
tests call `datasketches::hll_sketch::deserialize`, which is now stricter about
malformed input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
