xiangfu0 commented on PR #19309: URL: https://github.com/apache/pinot/pull/19309#issuecomment-5877731838
Reworked the integration coverage: one column per codec combination, across data types, with a raw uncompressed baseline per type. **68 matrix columns, one per (data type, codec) pair.** Every data type gets the `PASS_THROUGH` baseline — raw with no compression — plus `SNAPPY`, `ZSTANDARD`, `LZ4` and `GZIP`. INT and LONG additionally get 19 `codecSpec` pipelines each, which between them exercise all eight built-ins: each compression alone, `ZSTD(9)` for a non-default level, each transform alone (`DELTA`, `DELTADELTA`, `T64`, `GORILLA`), a chained value-preserving pair, a packing transform after a value-preserving one, transform+compression pairs covering all four compressions, two three-stage chains, and a multi-compression chain. `GORILLA,ZSTANDARD` also covers the legacy-spelling alias. **Why the other types have no `codecSpec`:** `ForwardIndexType.validateCodecPipelineShape` accepts it only for single-value INT/LONG, so FLOAT, DOUBLE, STRING and BYTES can only vary through the legacy `compressionCodec` path. `PASS_THROUGH` is also the only way to express "no compression" — the DSL has no identity codec. **The invariant you get from this:** for each data type, every column must read back identical data. That is asserted per type in a single query comparing all of that type's columns against its baseline across all 1000 rows, plus MIN/MAX aggregates and a 12-row spot check straddling chunk boundaries. BYTES compares via `bytesToHex` and FLOAT via `floatToIntBits`, since the two engines render those differently. **This does not cost more CI time than before.** Query count went the other way: 19 TestNG invocations and 42 query executions for 71 columns under test, versus 92 invocations for 14 specs previously. One query validates a whole type's worth of codecs rather than one query per codec, so the wide matrix is nearly free — the widest single query carries 23 predicates instead of 23 separate queries. `testGeneratedSegmentsUseConfiguredForwardIndexFormats` checks every column's on-disk format against its config: codecSpec columns must be the V7 reader with the canonical spec in the header, legacy columns must not be V7 and must report the expected `ChunkCompressionType`. The realtime test stays separate and small (3 columns) and covers only the consuming-to-committed conversion. Verified locally: all 38 emitted codecSpec strings accepted by the real validator with their canonical forms matching; V7 writer/reader round trips bit-exact for all 38 including `Long.MIN_VALUE`/`MAX_VALUE` through T64 bit-packing; real offline segments built from the generated Avro and all 68 columns x 1000 rows read back correctly; `TableConfigUtils.validate` passes on the generated config. The cluster suite itself needs CI. _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
