prtkgaur commented on code in PR #557: URL: https://github.com/apache/parquet-format/pull/557#discussion_r3676404735
########## Encodings.md: ########## @@ -391,3 +391,470 @@ After applying the transformation, the data has the following representation: ``` Bytes AA 00 A3 BB 11 B4 CC 22 C5 DD 33 D6 ``` + +<a name="ALP"></a> +### Adaptive Lossless floating-Point: (ALP = 10) + +Supported Types: FLOAT, DOUBLE + +This encoding is adapted from the paper +["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717) +by Afroozeh, Kuffo, and Boncz (SIGMOD 2024). + +ALP works by converting floating-point values to integers using decimal scaling +(controlled by an *exponent* `e` and *factor* `f`), then applying Frame of +Reference (FOR) encoding and bit-packing. Values that cannot be losslessly +converted are stored separately as *exceptions*. The encoding achieves high +compression for decimal-like floating-point data (e.g., monetary values, sensor +readings) while remaining fully lossless. Each value is encoded independently, +enabling random access to individual values and parallel encoding/decoding. + +#### Overview + +For each data page, ALP encoding consists of a header followed by an offset array +and one or more encoded vectors (batches of values). Each vector contains up to +`vector_size` elements (default 1024). + +``` ++-------------+-----------------------------+--------------------------------------+ +| Header | Offset Array | Vector Data | +| (7 bytes) | (num_vectors * 4 bytes) | (variable) | ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +| Page Header | off0 | off1 | ... | off N-1 | Vector 0 | Vector 1 | ... | Vec N-1 | +| (7 bytes) | (4B) | (4B) | | (4B) |(variable)|(variable)| |(variable)| ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +``` + +The compression pipeline below describes *one* way to produce a conforming +vector. It is informative, not normative: an encoder may use any strategy as long +as it emits the byte layout defined in [Page Layout](#page-layout). Only that +byte layout and the [Decoding](#decoding) procedure are normative. + +``` + Input: float/double array + | + v + +----------------------------------------------------------+ + | 1. CHOOSE PARAMETERS | + | Select (exponent, factor) pair for this array | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. DECIMAL ENCODING | + | encoded[i] = fast_round(value[i] * 10^e * 10^(-f)) | + | Detect exceptions where decode(encode(v)) != v | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. FRAME OF REFERENCE (FOR) | + | min_val = min(encoded[:]) | + | delta[i] = encoded[i] - min_val | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. BIT PACKING | + | bit_width = ceil(log2(max_delta + 1)) | + | Pack each delta into bit_width bits | + +----------------------------------------------------------+ + | + v + Output: Serialized vector bytes +``` + +The `fast_round` used in step 2 is one recommended rounding technique, described in +[Fast Rounding](#fast-rounding) below; it is informative, not normative. + +#### Page Layout + +##### Header (7 bytes) + +All multi-byte values are stored in little-endian order. + +``` + Byte: 0 1 2 3 4 5 6 + +----------------+---------------+--------------+----+----+----+----+ + | compression | integer | log_vector | num_elements | + | _mode | _encoding | _size | (int32 LE) | + +----------------+---------------+--------------+----+----+----+----+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). | +| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) | +| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) | +| 3 | num_elements | 4 bytes | int32 | Total number of non-null floating-point values in the page | + +The number of vectors is `ceil(num_elements / vector_size)`. The last vector may +contain fewer than `vector_size` elements. + +**Note:** The number of elements per vector is NOT stored in the header — it is +derived: `vector_size` for all vectors except the last, which may be smaller. + +##### Offset Array + +Immediately following the header is an array of `num_vectors` little-endian uint32 +values. Each offset gives the byte position of the corresponding vector's data, +measured from the start of the offset array itself. + +The first offset always equals `num_vectors * 4` (pointing just past the offset array). +Each subsequent offset equals the previous offset plus the stored size of the +previous vector. No padding is inserted between vectors. + +Offsets are relative to the start of the offset array. A vector's absolute byte +position is `alp_data_start + 7 + offset`, where `alp_data_start` is the first +byte of the ALP header within the *decoded* page data — that is, after the page +has been decompressed and after any repetition/definition levels — and `7` is the +size of the ALP header. (When the page is uncompressed and carries no repetition +or definition levels, `alp_data_start` coincides with the first byte after the +page's Thrift header.) + +##### Vector Format + +Each vector is self-describing and contains the encoding parameters, FOR metadata, +bit-packed encoded values, and exception data. The layout described here applies +when `compression_mode` = 0 (ALP) and `integer_encoding` = 0 (FOR + bit-packing); +future modes may define different vector contents and need not include `AlpInfo` +or `ForInfo`. + +``` +<----------- Vector Header -----------><----------------------- Data Section -----------------------> ++-------------------+-----------------+-------------------+---------------------+-------------------+ +| AlpInfo | ForInfo | PackedValues | ExceptionPositions | ExceptionValues | +| (4 bytes) | (5B or 9B) | (variable) | (variable) | (variable) | ++-------------------+-----------------+-------------------+---------------------+-------------------+ +``` + +The first two components (`AlpInfo` and `ForInfo`) form the *vector header*; the +remaining three (`PackedValues`, `ExceptionPositions`, `ExceptionValues`) form the +*data section*. + +Vector header sizes: +| Type | AlpInfo | ForInfo | Total Header | +|--------|---------|---------|--------------| +| FLOAT | 4 bytes | 5 bytes | 9 bytes | +| DOUBLE | 4 bytes | 9 bytes | 13 bytes | + +Data section sizes: +| Section | Size Formula | Description | +|---------------------|-----------------------------|------------------------------| +| PackedValues | ceil(num\_elements\_in\_vector * bit\_width / 8) | Bit-packed delta values | +| ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | +| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original values, stored as their exact IEEE-754 bits (NaN not canonicalized) | + +Here `bit_width` and `num_exceptions` are read from the vector header (`ForInfo` +and `AlpInfo` respectively), described below. + +###### AlpInfo (4 bytes, both types) + +``` + Byte: 0 1 2 3 + +----------+----------+---------+---------+ + | exponent | factor | num_exceptions | + | (uint8) | (uint8) | (uint16 LE) | + +----------+----------+---------+---------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | exponent | 1 byte | uint8 | Power-of-10 exponent *e*. Range: \[0, 10\] for FLOAT, \[0, 18\] for DOUBLE. | +| 1 | factor | 1 byte | uint8 | Power-of-10 factor *f*. Range: \[0, *e*\]. | +| 2 | num_exceptions | 2 bytes | uint16 | Number of exception values in this vector. | + +###### ForInfo for FLOAT (5 bytes) + +``` + Byte: 0 1 2 3 4 + +----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int32 LE) | (uint8) | + +----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 4 bytes | int32 | Minimum encoded integer in the vector | +| 4 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 32\]. | + +###### ForInfo for DOUBLE (9 bytes) + +``` + Byte: 0 1 2 3 4 5 6 7 8 + +----+----+----+----+----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int64 LE) | (uint8) | + +----+----+----+----+----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 8 bytes | int64 | Minimum encoded long in the vector | +| 8 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 64\]. | + +###### PackedValues + +The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes. +Values are bit-packed using the same LSB-first packing order as the +[RLE/Bit-Packing Hybrid](#RLE) encoding. When the total number of packed bits is +not a multiple of 8, the final byte is padded with zero bits in its most +significant positions. + +Each delta is `encoded[i] - frame_of_reference`, computed in unsigned (wrapping) +arithmetic and stored as an unsigned integer. Computing it as unsigned avoids +signed-integer overflow when the vector's range (`max - min`) exceeds the signed +maximum of the encoded type, and it means no sign extension is applied when +unpacking. Because `frame_of_reference` is the minimum encoded integer in the +vector, every delta is non-negative. + +If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded +integers are equal to `frame_of_reference`). + +###### ExceptionPositions + +An array of `num_exceptions` little-endian uint16 values, each giving +the 0-based index within the vector of an exception value. + +###### ExceptionValues + +An array of `num_exceptions` values in the original floating-point type +(4 bytes little-endian IEEE 754 for FLOAT, 8 bytes for DOUBLE), stored in +the same order as the corresponding positions. Each value is stored as its exact +IEEE 754 bit pattern; implementations MUST NOT canonicalize NaN or otherwise alter +the bits, so that decoding reproduces the original value bit-for-bit. + +#### Encoding + +##### Encoding Formula + +``` ++-------------------------------------------------------------------+ +| | +| encoded = fast_round( value * 10^e * 10^(-f) ) | +| | +| decoded = encoded * 10^f * 10^(-e) | +| | ++-------------------------------------------------------------------+ +``` + +The formula uses two separate multiplications (not a single multiplication by +`10^(e-f)`, and not division). This is a requirement of the **decode** path, which +is normative: to reconstruct a value every reader MUST compute +`decoded = encoded * 10^f * 10^(-e)` using the same two-step multiplication and the +same power-of-10 constants, so that all implementations reproduce the stored value +bit-for-bit. The power-of-10 constants MUST be the correctly-rounded IEEE 754 +values of the decimal literals `1e0`, `1e1`, ..., `1e18` and `1e-1`, `1e-2`, ..., +`1e-18` as defined by the decimal-to-binary conversion in IEEE 754-2008 §5.12.2. +Implementations MUST NOT compute these constants at runtime via `pow()` or +equivalent functions, which are not guaranteed to be correctly rounded. + +The **encode** direction — mapping each value to the integer it will be stored as, +via `fast_round(value * 10^e * 10^(-f))` — is informative, not normative. An +encoder MAY choose that integer by any means, because every value is checked +against the normative decode above and any value that does not round-trip exactly +is stored as an exception. The rounding method therefore affects only compression +ratio and exception count, never correctness or what a reader decodes. The +`fast_round` technique below is one recommended implementation. + +##### Fast Rounding (informative) Review Comment: @alamb do you recommend I simplify the language here or remove part of the section? ########## Encodings.md: ########## @@ -391,3 +391,470 @@ After applying the transformation, the data has the following representation: ``` Bytes AA 00 A3 BB 11 B4 CC 22 C5 DD 33 D6 ``` + +<a name="ALP"></a> +### Adaptive Lossless floating-Point: (ALP = 10) + +Supported Types: FLOAT, DOUBLE + +This encoding is adapted from the paper +["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717) +by Afroozeh, Kuffo, and Boncz (SIGMOD 2024). + +ALP works by converting floating-point values to integers using decimal scaling +(controlled by an *exponent* `e` and *factor* `f`), then applying Frame of +Reference (FOR) encoding and bit-packing. Values that cannot be losslessly +converted are stored separately as *exceptions*. The encoding achieves high +compression for decimal-like floating-point data (e.g., monetary values, sensor +readings) while remaining fully lossless. Each value is encoded independently, +enabling random access to individual values and parallel encoding/decoding. + +#### Overview + +For each data page, ALP encoding consists of a header followed by an offset array +and one or more encoded vectors (batches of values). Each vector contains up to +`vector_size` elements (default 1024). + +``` ++-------------+-----------------------------+--------------------------------------+ +| Header | Offset Array | Vector Data | +| (7 bytes) | (num_vectors * 4 bytes) | (variable) | ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +| Page Header | off0 | off1 | ... | off N-1 | Vector 0 | Vector 1 | ... | Vec N-1 | +| (7 bytes) | (4B) | (4B) | | (4B) |(variable)|(variable)| |(variable)| ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +``` + +The compression pipeline below describes *one* way to produce a conforming +vector. It is informative, not normative: an encoder may use any strategy as long +as it emits the byte layout defined in [Page Layout](#page-layout). Only that +byte layout and the [Decoding](#decoding) procedure are normative. + +``` + Input: float/double array + | + v + +----------------------------------------------------------+ + | 1. CHOOSE PARAMETERS | + | Select (exponent, factor) pair for this array | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. DECIMAL ENCODING | + | encoded[i] = fast_round(value[i] * 10^e * 10^(-f)) | + | Detect exceptions where decode(encode(v)) != v | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. FRAME OF REFERENCE (FOR) | + | min_val = min(encoded[:]) | + | delta[i] = encoded[i] - min_val | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. BIT PACKING | + | bit_width = ceil(log2(max_delta + 1)) | + | Pack each delta into bit_width bits | + +----------------------------------------------------------+ + | + v + Output: Serialized vector bytes +``` + +The `fast_round` used in step 2 is one recommended rounding technique, described in +[Fast Rounding](#fast-rounding) below; it is informative, not normative. + +#### Page Layout + +##### Header (7 bytes) + +All multi-byte values are stored in little-endian order. + +``` + Byte: 0 1 2 3 4 5 6 + +----------------+---------------+--------------+----+----+----+----+ + | compression | integer | log_vector | num_elements | + | _mode | _encoding | _size | (int32 LE) | + +----------------+---------------+--------------+----+----+----+----+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). | +| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) | +| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) | +| 3 | num_elements | 4 bytes | int32 | Total number of non-null floating-point values in the page | + +The number of vectors is `ceil(num_elements / vector_size)`. The last vector may +contain fewer than `vector_size` elements. + +**Note:** The number of elements per vector is NOT stored in the header — it is +derived: `vector_size` for all vectors except the last, which may be smaller. + +##### Offset Array + +Immediately following the header is an array of `num_vectors` little-endian uint32 +values. Each offset gives the byte position of the corresponding vector's data, +measured from the start of the offset array itself. + +The first offset always equals `num_vectors * 4` (pointing just past the offset array). +Each subsequent offset equals the previous offset plus the stored size of the +previous vector. No padding is inserted between vectors. + +Offsets are relative to the start of the offset array. A vector's absolute byte +position is `alp_data_start + 7 + offset`, where `alp_data_start` is the first +byte of the ALP header within the *decoded* page data — that is, after the page +has been decompressed and after any repetition/definition levels — and `7` is the +size of the ALP header. (When the page is uncompressed and carries no repetition +or definition levels, `alp_data_start` coincides with the first byte after the +page's Thrift header.) + +##### Vector Format + +Each vector is self-describing and contains the encoding parameters, FOR metadata, +bit-packed encoded values, and exception data. The layout described here applies +when `compression_mode` = 0 (ALP) and `integer_encoding` = 0 (FOR + bit-packing); +future modes may define different vector contents and need not include `AlpInfo` +or `ForInfo`. + +``` +<----------- Vector Header -----------><----------------------- Data Section -----------------------> ++-------------------+-----------------+-------------------+---------------------+-------------------+ +| AlpInfo | ForInfo | PackedValues | ExceptionPositions | ExceptionValues | +| (4 bytes) | (5B or 9B) | (variable) | (variable) | (variable) | ++-------------------+-----------------+-------------------+---------------------+-------------------+ +``` + +The first two components (`AlpInfo` and `ForInfo`) form the *vector header*; the +remaining three (`PackedValues`, `ExceptionPositions`, `ExceptionValues`) form the +*data section*. + +Vector header sizes: +| Type | AlpInfo | ForInfo | Total Header | +|--------|---------|---------|--------------| +| FLOAT | 4 bytes | 5 bytes | 9 bytes | +| DOUBLE | 4 bytes | 9 bytes | 13 bytes | + +Data section sizes: +| Section | Size Formula | Description | +|---------------------|-----------------------------|------------------------------| +| PackedValues | ceil(num\_elements\_in\_vector * bit\_width / 8) | Bit-packed delta values | +| ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | +| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original values, stored as their exact IEEE-754 bits (NaN not canonicalized) | + +Here `bit_width` and `num_exceptions` are read from the vector header (`ForInfo` +and `AlpInfo` respectively), described below. + +###### AlpInfo (4 bytes, both types) + +``` + Byte: 0 1 2 3 + +----------+----------+---------+---------+ + | exponent | factor | num_exceptions | + | (uint8) | (uint8) | (uint16 LE) | + +----------+----------+---------+---------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | exponent | 1 byte | uint8 | Power-of-10 exponent *e*. Range: \[0, 10\] for FLOAT, \[0, 18\] for DOUBLE. | +| 1 | factor | 1 byte | uint8 | Power-of-10 factor *f*. Range: \[0, *e*\]. | +| 2 | num_exceptions | 2 bytes | uint16 | Number of exception values in this vector. | + +###### ForInfo for FLOAT (5 bytes) + +``` + Byte: 0 1 2 3 4 + +----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int32 LE) | (uint8) | + +----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 4 bytes | int32 | Minimum encoded integer in the vector | +| 4 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 32\]. | + +###### ForInfo for DOUBLE (9 bytes) + +``` + Byte: 0 1 2 3 4 5 6 7 8 + +----+----+----+----+----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int64 LE) | (uint8) | + +----+----+----+----+----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 8 bytes | int64 | Minimum encoded long in the vector | +| 8 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 64\]. | + +###### PackedValues + +The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes. +Values are bit-packed using the same LSB-first packing order as the +[RLE/Bit-Packing Hybrid](#RLE) encoding. When the total number of packed bits is +not a multiple of 8, the final byte is padded with zero bits in its most +significant positions. + +Each delta is `encoded[i] - frame_of_reference`, computed in unsigned (wrapping) +arithmetic and stored as an unsigned integer. Computing it as unsigned avoids +signed-integer overflow when the vector's range (`max - min`) exceeds the signed +maximum of the encoded type, and it means no sign extension is applied when +unpacking. Because `frame_of_reference` is the minimum encoded integer in the +vector, every delta is non-negative. + +If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded +integers are equal to `frame_of_reference`). + +###### ExceptionPositions + +An array of `num_exceptions` little-endian uint16 values, each giving +the 0-based index within the vector of an exception value. + +###### ExceptionValues + +An array of `num_exceptions` values in the original floating-point type +(4 bytes little-endian IEEE 754 for FLOAT, 8 bytes for DOUBLE), stored in +the same order as the corresponding positions. Each value is stored as its exact +IEEE 754 bit pattern; implementations MUST NOT canonicalize NaN or otherwise alter +the bits, so that decoding reproduces the original value bit-for-bit. + +#### Encoding + +##### Encoding Formula + +``` ++-------------------------------------------------------------------+ +| | +| encoded = fast_round( value * 10^e * 10^(-f) ) | +| | +| decoded = encoded * 10^f * 10^(-e) | +| | ++-------------------------------------------------------------------+ +``` + +The formula uses two separate multiplications (not a single multiplication by +`10^(e-f)`, and not division). This is a requirement of the **decode** path, which +is normative: to reconstruct a value every reader MUST compute +`decoded = encoded * 10^f * 10^(-e)` using the same two-step multiplication and the +same power-of-10 constants, so that all implementations reproduce the stored value +bit-for-bit. The power-of-10 constants MUST be the correctly-rounded IEEE 754 +values of the decimal literals `1e0`, `1e1`, ..., `1e18` and `1e-1`, `1e-2`, ..., +`1e-18` as defined by the decimal-to-binary conversion in IEEE 754-2008 §5.12.2. +Implementations MUST NOT compute these constants at runtime via `pow()` or +equivalent functions, which are not guaranteed to be correctly rounded. + +The **encode** direction — mapping each value to the integer it will be stored as, +via `fast_round(value * 10^e * 10^(-f))` — is informative, not normative. An +encoder MAY choose that integer by any means, because every value is checked +against the normative decode above and any value that does not round-trip exactly +is stored as an exception. The rounding method therefore affects only compression +ratio and exception count, never correctness or what a reader decodes. The +`fast_round` technique below is one recommended implementation. + +##### Fast Rounding (informative) Review Comment: I'm going for simplification for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
