leerho opened a new pull request, #770: URL: https://github.com/apache/datasketches-java/pull/770
Java counterpart of apache/datasketches-cpp#527 and apache/datasketches-cpp#529. ### Problem `isEmpty()` in `FrequentLongsSketch` and `FrequentItemsSketch` was `getNumActiveItems() == 0`. A purge subtracts the median from every counter and drops those that reach zero, so with many low-frequency items it can remove every counter while `streamWeight` and `offset` stay positive. That state was reported as empty: - `toByteArray()` wrote the 8-byte empty form, dropping stream weight and offset. After a round trip the sketch looked fresh and exact. - `merge()` returned early and ignored the other sketch's stream weight and offset. Reproducer: `new FrequentLongsSketch(256)` updated with 193 distinct items. Before this change it reports `isEmpty() == true` with stream length 193 and error 1; after a byte round trip, stream length 0 and error 0; merged into a sketch with one item, stream length 1 and error 0. A non-empty sketch with no items is not equivalent to a fresh one: it says every item's count is at most `offset`, and its stream length counts. ### Changes - **`isEmpty()` returns `streamWeight == 0`** in both classes. Updates only accept positive counts and merges only add, so only `reset()` returns a sketch to empty. - **`toByteArray()` handles zero active items** in a non-empty sketch: full preamble with `activeItems = 0` and no values or keys (32 bytes). This path was unreachable before, and the hash maps return `null` active arrays when empty. - **Reading images: PreLongs decides emptiness** (1 empty, 4 non-empty). The empty flag (mask 0x05) is only cross-checked. A non-empty image whose stream weight is not positive is rejected as corrupt. - **`FrequentLongsSketch` string form**: `serializeToString()` sets the flag from `isEmpty()`; `getInstance(String)` masks the flag with `EMPTY_FLAG_MASK` (was `flags > 0`), requires it to agree with the stream weight, and no longer rejects a non-empty sketch with no active items. - **`PreambleUtil`**: documented emptiness and the flags byte (both legacy bits written, either accepted, no other bits defined), fixed the row label of the data start in the layout diagram, and removed the unused `SER_DE_ID_SHORT`. No change to the binary format. Every image written by an earlier version still deserializes; purged-to-zero images written earlier were already stored as empty, and their stream weight and offset cannot be recovered. ### Tests - Purged-to-zero sketch, longs and items: not empty, 32-byte image with PreLongs 4 and flags 0, byte and string round trips, merge keeps stream weight and error. - `reset()` after a purge returns to the empty 8-byte form. - Empty image accepted with flags 0x01, 0x04 and 0x05. - Corrupt images rejected: PreLongs 1 without the empty flag, PreLongs 4 with it, zero stream weight; string form with the flag disagreeing with the stream weight. - `LongsSketchTest.checkStringDeserEmptyCorrupt` asserted the old rule (its input is a valid purged-to-zero sketch); it is now `checkStringDeserNonEmptyNoItems`. ### Cross-language - `FrequentItemsSketchCrossLanguageTest` generates `frequent_long_purged_java.sk` and `frequent_string_purged_java.sk` (`lgMaxMapSize = 8`, 193 distinct items) and checks them for Java, C++ and Go. The C++ images come from apache/datasketches-cpp#529; the Java and C++ images are byte-identical for both longs and strings. - `checkCpp()` called `stringsAscii` and `stringsUtf8` with `GroupLanguage.JAVA`, so it never read the C++ ascii and utf8 images. It now uses `GroupLanguage.CPP`. Go reads these images too; `checkGo()` will expose the same bug in datasketches-go once it generates the purged images. ### Unrelated failures `BloomFilterTest.basicDifferenceTest` and `TDigestDoubleSerializationTest.weightOverflowDoesNotChangeDigest` fail on `main` without this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
