This is an automated email from the ASF dual-hosted git repository. leerho pushed a commit to branch tuple-preamble-util in repository https://gitbox.apache.org/repos/asf/datasketches-java.git
commit a784416a2e734799d798560487811122e76236f5 Author: Lee Rhodes <[email protected]> AuthorDate: Thu Oct 29 13:26:16 2026 -0700 Tuple: add PreambleUtil documenting the serialized forms Documents the compact (serial version 3 and legacy 1, 2) and updatable (QuickSelectSketch) layouts, the flags byte of each form, and the encodings of the example Summaries, in one place, following theta's PreambleUtil. Adds the byte-address and serial-version constants. No behavior change. Co-Authored-By: Claude Opus 5.5 <[email protected]> --- .../apache/datasketches/tuple/PreambleUtil.java | 195 +++++++++++++++++++++ 1 file changed, 195 insertions(+) diff --git a/src/main/java/org/apache/datasketches/tuple/PreambleUtil.java b/src/main/java/org/apache/datasketches/tuple/PreambleUtil.java new file mode 100644 index 000000000..412afb936 --- /dev/null +++ b/src/main/java/org/apache/datasketches/tuple/PreambleUtil.java @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datasketches.tuple; + +//@formatter:off + +/** + * This class documents the serialized forms of the generic tuple sketches, + * {@link CompactTupleSketch} and the updatable {@link QuickSelectSketch}, in one place. + * The ArrayOfDoubles sketches in the <i>arrayofdoubles</i> sub-package have their own layouts, + * which are documented in those classes. + * + * <p>MAP: Low significance bytes of this <i>long</i> data structure are on the right. + * All multi-byte values (<i>short</i>, <i>int</i>, <i>long</i>, <i>float</i> and <i>double</i>) + * are stored little-endian. The <i>byte</i> values are treated as unsigned.</p> + * + * <p>Unlike the theta sketches, a tuple sketch image is not a fixed-size table of longs. + * Each retained entry is a hash (8 bytes) followed by the bytes of its Summary, and a Summary + * may have any length. The Summary bytes are written by {@link Summary#toByteArray()} and read by a + * {@link SummaryDeserializer}, both supplied by the user. The sketch image does not record the + * Summary type. A reader must therefore know which Summary was used to write the image.</p> + * + * <h2>CompactTupleSketch, serial version 3 (current)</h2> + * + * <p>This form is binary compatible with the C++ and Go compact tuple sketches, as long as + * every language uses the same Summary encoding.</p> + * + * <p>Flags: ReadOnly, Compact, Empty if empty. The Ordered flag is defined but Java does not set it. + * The seed hash is zero when the sketch is empty, as in theta.</p> + * + * <p>An empty sketch, or a single-item sketch that is not in estimation mode, uses 1 preamble long. + * The single entry, if any, starts at byte 8.</p> + * + * <pre> + * Long || Start Byte Adr: + * Adr: + * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | + * 0 || Seed Hash | Flags | unused | SkType | FamID | SerVer | PreLongs = 1 | + * + * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | + * 1 ||-------------------------Hash of the single entry (if not empty)-------------------| + * + * || | | | | | | | 16 | + * 2 ||-------------------------Summary of the single entry-------------------------------| + * </pre> + * + * <p>An exact (not estimating) sketch with more than one entry uses 2 preamble longs.</p> + * + * <pre> + * Long || Start Byte Adr: + * Adr: + * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | + * 0 || Seed Hash | Flags | unused | SkType | FamID | SerVer | PreLongs = 2 | + * + * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | + * 1 ||-------------unused----------------|----------Retained Entries Count---------------| + * + * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | + * 2 ||-------------Entries: hash (8 bytes), then Summary bytes, repeated Count times-----| + * </pre> + * + * <p>An estimating sketch uses 3 preamble longs.</p> + * + * <pre> + * Long || Start Byte Adr: + * Adr: + * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | + * 0 || Seed Hash | Flags | unused | SkType | FamID | SerVer | PreLongs = 3 | + * + * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | + * 1 ||-------------unused----------------|----------Retained Entries Count---------------| + * + * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | + * 2 ||------------------------------THETA_LONG-------------------------------------------| + * + * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | + * 3 ||-------------Entries: hash (8 bytes), then Summary bytes, repeated Count times-----| + * </pre> + * + * <h2>CompactTupleSketch, serial versions 1 and 2 (legacy, read only)</h2> + * + * <p>Java no longer writes these forms but can still read them. The fields after byte 4 are + * packed with no alignment, and each one is present only if its flag is set.</p> + * + * <pre> + * Byte Adr | Size | Field + * 0 | 1 | PreLongs + * 1 | 1 | SerVer = 1 or 2 + * 2 | 1 | FamID + * 3 | 1 | SkType = CompactTupleSketch + * 4 | 1 | Flags (legacy bits, see below) + * 5 | 8 | THETA_LONG, if IS_THETA_INCLUDED + * next | 1 | Summary class-name length N, if HAS_ENTRIES and SerVer = 1 + * next | 4 | Retained Entries Count, if HAS_ENTRIES + * next | N | Summary class name, if HAS_ENTRIES and SerVer = 1 (skipped) + * next | 8*C | Count hashes, if HAS_ENTRIES + * next | var | Count Summaries, if HAS_ENTRIES + * </pre> + * + * <h2>QuickSelectSketch (updatable), serial version 2 (deprecated)</h2> + * + * <p>Serializing an updatable sketch is deprecated. This form has no seed hash and is + * read only by Java. As with the legacy compact form, the fields after byte 7 are + * packed, and each one is present only if its flag is set.</p> + * + * <pre> + * Long || Start Byte Adr: + * Adr: + * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | + * 0 || LgRF | LgArr | LgNom | Flags | SkType | FamID | SerVer | PreLongs = 1 | + * + * Byte Adr | Size | Field + * 8 | 4 | Sampling probability p (float), if IS_IN_SAMPLING_MODE + * next | 8 | THETA_LONG, if IS_THETA_INCLUDED + * next | 4 | Retained Entries Count, if HAS_ENTRIES + * next | var | Entries: hash (8 bytes), then Summary bytes, repeated Count times + * </pre> + * + * <h2>Flags byte</h2> + * + * <pre> + * Bit | CompactTupleSketch v3 | CompactTupleSketch v1, v2 | QuickSelectSketch v2 + * 0 | Reserved | IS_BIG_ENDIAN (unused) | Reserved + * 1 | IS_READ_ONLY | IS_EMPTY | IS_IN_SAMPLING_MODE + * 2 | IS_EMPTY | HAS_ENTRIES | IS_EMPTY + * 3 | IS_COMPACT | IS_THETA_INCLUDED | HAS_ENTRIES + * 4 | IS_ORDERED | | IS_THETA_INCLUDED + * </pre> + * + * <h2>Summary encodings of the example Summaries in this library</h2> + * + * <p>The Summaries in the <i>adouble</i>, <i>aninteger</i> and <i>strings</i> sub-packages are + * examples of user-defined Summaries. Their encodings are specific to Java, and other languages + * can read them only through custom code. The Mode byte is the ordinal of the Summary's Mode: + * Sum = 0, Min = 1, Max = 2, AlwaysOne = 3.</p> + * + * <pre> + * Summary | Bytes | Encoding + * DoubleSummary | 9 | double value (bytes 0-7), Mode (byte 8) + * IntegerSummary | 5 | int value (bytes 0-3), Mode (byte 4) + * ArrayOfStringsSummary | var | int total bytes, byte number of strings, + * | | then for each string: int length, then its UTF-8 bytes + * </pre> + * + * @author Lee Rhodes + */ +final class PreambleUtil { + + private PreambleUtil() {} + + // Byte addresses common to all forms + static final int PREAMBLE_LONGS_BYTE = 0; + static final int SER_VER_BYTE = 1; + static final int FAMILY_BYTE = 2; + static final int SKETCH_TYPE_BYTE = 3; + + // CompactTupleSketch, serial version 3 + static final int COMPACT_FLAGS_BYTE = 5; + static final int SEED_HASH_SHORT = 6; //bytes 6,7 + static final int RETAINED_ENTRIES_INT = 8; //only if PreLongs > 1 + static final int THETA_LONG = 16; //only if PreLongs = 3 + + // CompactTupleSketch, serial versions 1 and 2, and QuickSelectSketch + static final int LEGACY_FLAGS_BYTE = 4; + + // QuickSelectSketch, serial version 2 + static final int LG_NOM_ENTRIES_BYTE = 5; + static final int LG_CUR_CAPACITY_BYTE = 6; + static final int LG_RESIZE_FACTOR_BYTE = 7; + static final int QS_VARIABLE_START = 8; + + // Serial versions + static final byte COMPACT_SER_VER_WITH_CLASS_NAME = 1; + static final byte COMPACT_SER_VER_LEGACY = 2; + static final byte COMPACT_SER_VER = 3; + static final byte QUICK_SELECT_SER_VER = 2; + +} --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
