xiangfu0 opened a new pull request, #19309:
URL: https://github.com/apache/pinot/pull/19309
Stacked on #19308 (`xiangfu0/codex/codec-stack/06-reload-enable`). Part of
the split of #18229.
## Summary
- **`CodecPipelineIntegrationTest`** (pinot-integration-tests, `custom/`
suite): end-to-end
coverage of `indexes.forward.codecSpec` on an offline table. Every
supported codec spec gets its
own INT and LONG raw column, all populated with identical values, so every
codec must read back
the same aggregates, filter counts, and point lookups on both query
engines (SSE + MSE):
- Single-stage compression: `LZ4`, `ZSTD(3)`, `SNAPPY`, `GZIP`
(legacy-compatible raw writer path)
- Transform + compression chains: `DELTA,LZ4`, `DELTA,ZSTD(3)`,
`DELTADELTA,LZ4` (V7 writer path)
- Packing transforms: `T64`, `GORILLA`, `T64,LZ4`, `GORILLA,ZSTD(3)` (V7
writer path)
- Per-codec SUM/filter checks, point lookups across real V7 chunk
boundaries
(`targetDocsPerChunk=256` against 500-row input files), a cross-codec
same-row consistency
query, coexistence with a dictionary-encoded STRING column, and a RAW
STRING column using a
compression-only `codecSpec` (`ZSTD(3)`) alongside V7 transform columns
in the same segment.
- **`docs/design/codec-pipeline-v7.md`**: design doc for the codec pipeline
framework — DSL
grammar (matching `CodecSpecParser`), architecture, V7 on-disk format and
reader-side corruption
defenses, built-in codec catalog, configuration and validation, user
manual (enable / migrate /
roll back / mixed-version safety / common errors), threading, performance,
backward
compatibility, test-coverage map, and future work. Placement under
`docs/design/` is noted as
subject to reviewer preference.
All spec strings use the wrapper-less comma-list DSL introduced by this
series (`DELTA,T64,LZ4`,
`ZSTD(3)`). The doc mentions the removed `CODEC(...)` wrapper only to
document that the parser
rejects it with a precise error.
## Exclusions relative to the original #18229
- `CompressionCodecMigrator` (and its tests) were dropped from the series;
the design doc refers
to future migration tooling instead and keeps the manual
`compressionCodec` → `codecSpec`
migration table.
- No test in this slice depends on migration helpers; all table configs are
built directly with
`indexes.forward.codecSpec`.
## Safety
Test-and-docs-only slice: no production code changes. The integration test
only exercises code
paths introduced by the earlier slices of this series (03–06). No wire
formats, configs, or public
APIs are touched.
## Verification
```
./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl
pinot-integration-tests -am
./mvnw -q test -Ppinot-fastdev -pl pinot-integration-tests \
-Dtest=CodecPipelineIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format checkstyle:check license:check -pl
pinot-integration-tests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
