HappenLee opened a new pull request, #69217:
URL: https://github.com/apache/doris/pull/69217
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Zone maps already collect typed extrema, but serialize numeric and temporal
bounds as text and parse them again when loading indexes. This adds
format-conversion work for every page and segment, and makes large integer and
high-precision decimal bounds unnecessarily long.
Add a shared fixed-width little-endian codec for 19 persisted scalar types:
BOOLEAN, the five signed integer types, FLOAT/DOUBLE, Decimal32/64/128/256,
DATEV2/DATETIMEV2, TIMESTAMPTZ, TIMESTAMP_NS, IPv4/IPv6 and UUID. Page and
segment writers share the codec. String bounds retain their existing encoding
and truncation rules. Legacy DATE, DATETIME and DECIMALV2 retain text bounds
and do not gain binary codecs. Decoding restores existing Field values for
pruning, expression zone maps and metadata MIN/MAX; it does not compare encoded
bytes.
Compatibility:
- Add optional ZoneMapPB field 9 (`value_encoding`), checked against
branch-3.1, branch-4.0, branch-4.1, branch-4.2 and master. Absent/0 means
existing text; 1 means binary. New readers accept both formats independently of
the write setting. Unknown encodings, unsupported binary types and malformed
binary lengths fail explicitly.
- `enable_binary_zone_map` defaults to false. Each writer captures the
setting at construction for both index levels. Keep it disabled throughout a
rolling upgrade and enable it only after every reader supports the new format.
- Old BE versions cannot read binary bounds. Disabling the setting does not
convert existing files or change existing writers. Before downgrading, finish
those writers and rewrite every relevant binary file with a new reader and
legacy writes, including clone/backup dependencies.
NULL, pass_all, NaN/Inf flags and existing legacy-float safety behavior are
preserved. Fixed width is not always smaller for small integers; the included
design records that limit and the format/upgrade details in
`docs/design/zonemap-binary-encoding.md`. No end-to-end performance gain is
claimed.
### Release note
Add opt-in binary ZoneMap bounds for 19 fixed-width scalar types. Binary
writes remain disabled by default and require all readers to be upgraded before
enabling.
### Check List (For Author)
- Test
- [x] Unit Test: ASAN `run-be-ut.sh -j48 --run
--filter=ColumnZoneMapTest.*` — 39/39 passed.
- File-backed page/segment round trips and EQ pruning for all 19 types
and nullable/non-nullable schemas; canonical bytes, Decimal256 limbs, negative
nanoseconds, special floats, corrupt input, historical protobuf bytes,
mixed-format reads independent of configuration, and legacy-type text fallback.
- clang-format 16.0.6, build hygiene, changed-line clang-tidy and `git
diff --check`: passed.
- The added codec benchmark passes a C++ syntax check with the actual UT
compiler flags. Full SQL cluster regression, live mixed-version
upgrade/downgrade and RELEASE performance benchmarks were not run.
- Behavior changed:
- [x] Yes. Opt-in persisted bounds use binary encoding; default writes
remain in the existing format.
- Does this need documentation?
- [x] Yes. Design, type layout, compatibility and upgrade/downgrade
procedure included in `docs/design/zonemap-binary-encoding.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
