xiangfu0 opened a new pull request, #19306:
URL: https://github.com/apache/pinot/pull/19306
Stacked on #19305 (`xiangfu0/codex/codec-stack/03-delta-codecs`). Part of
the split of #18229.
## Summary
Adds the two **packing transform** codecs to the package-private codec
runtime in
`pinot-segment-local` `org.apache.pinot.segment.local.io.codec`:
- **`T64CodecDefinition`** — frame-of-reference (FOR) bit-packing over fixed
64-value blocks for
SV INT/LONG. Each block stores `[baseline][bitWidth][bit-packed deltas]`
inside a
`[flag][count]` frame. Partial last blocks still emit full 64-slot packed
payloads (zero-filled
tail) so payload size scales with block count.
- **`GorillaCodecDefinition`** — Gorilla-style XOR compression (Facebook
VLDB 2015) for SV
INT/LONG: verbatim first value, then per-value `0` bit for repeats or
`1 + window metadata + meaningful bits`, with window reuse between
consecutive XOR deltas.
Both are `CodecKind.TRANSFORM` stages with `preservesTypedValueLayout() ==
false`: they consume a
typed value array and emit a headered byte frame, so the existing
`CodecPipelineValidator`
structural rule automatically enforces that they must be the **last
transform** (only compression
stages may follow). No validator changes were needed — the rule generalizes
from the synthetic
`PACKING` handler introduced in PR 02.
Both are registered in `CodecRegistry.DEFAULT` (names `T64` / `GORILLA` are
frozen on-disk
contract strings). Both codecs are corrupt-segment defensive: header
flag/count validation runs
before any count-based short-circuit, bit widths and window `(leading,
width)` pairs are
range-checked, and `decodeInto` enforces dst-capacity bounds.
## Tests
- `T64CodecDefinitionTest` — round-trips for
empty/single/all-equal/partial-block/max-range
inputs; a parameterized sweep over every bitWidth 1..32 (INT) and 1..64
(LONG); an on-disk
byte-layout pin test; encoded-size-within-`maxEncodedSize` bound checks;
corrupt-segment
defenses (bad flag, bad bitWidth, negative count, frame/column type
mismatch).
- `GorillaCodecDefinitionTest` — round-trips for
repeat/window-reuse/explicit-window transitions,
boundary values, timestamp/counter-like sequences; corrupt-segment
defenses including
window-reuse-before-explicit.
- `CodecPipelineValidatorTest` — real-codec packing-rule scenarios through
`CodecRegistry.DEFAULT`: `DELTA,T64,LZ4` / `DELTADELTA,GORILLA,ZSTD(3)` /
`DELTA,T64` accepted;
compression-after-packing (`T64,ZSTD(3)`, `GORILLA,SNAPPY`,
`T64,LZ4,GZIP`) accepted;
`T64,DELTA` and `T64,GORILLA` rejected; type restrictions and no-arg
enforcement;
`preservesTypedValueLayout()` contract pinned to `false` for both.
- `CodecRegistryTest` — `T64`/`GORILLA` now resolve from the default
registry (case-insensitive).
- `TableConfigUtilsTest` — a mis-ordered `T64,DELTA,LZ4` codecSpec is still
gate-rejected at
config time with `codecSpec is not supported yet for column: intCol`.
## Deliberately excluded (carried by later PRs)
- **V7 raw forward index format** (writer/reader that actually executes
codec pipelines against
segment chunks) — next PR in the stack (05, from local
`codec-split/07-v7-format`).
- **Opening the feature gate** (config/creator/reader/reload surfaces
accepting codecSpec,
reload change-detection via canonical specs) — final PR (from local
`codec-split/08-reload-enable`).
- Any public SPI exposure: both codecs are package-private inside
`io.codec`; the only
cross-package entry point remains `CodecPipelineExecutor`.
## Why master stays safe with only this merged
The codecSpec feature gate is untouched and still closed: every
config/creator/reader/reload
surface continues to throw `codecSpec is not supported yet for column: %s`,
so no production
path can reach these codecs. The new classes are package-private additions
to `io.codec` that are
only reachable through `CodecPipelineExecutor`, which nothing in production
instantiates yet.
No on-disk format, wire protocol, or public API changes.
## Verification
```
./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-segment-local
-am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
-Dtest=T64CodecDefinitionTest,GorillaCodecDefinitionTest,CodecPipelineValidatorTest,CodecRegistryTest,DeltaCodecRoundTripTest,CodecPipelineExecutorTest
\
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local
-Dtest=TableConfigUtilsTest \
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-segment-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
