raghavyadav01 opened a new pull request, #19608:
URL: https://github.com/apache/pinot/pull/19608
Stacked on #19607 (the declared-child-spec fix), which this needs for the
child's single/multi-value flag. Review that one first; the second commit here
is the multi-value work.
### The problem
A key whose values are collections has nowhere to go.
`OpenStructTypeInference` does not map a collection, so the key falls back to
STRING and the column stores the list's `toString`:
```
{"tags": [1, 2, 3]} → props$tags = "[1, 2, 3]" // not even JSON
```
Nothing inside it can be filtered, counted or grouped, and the elements' own
types are gone.
### Change
Materialize such a key as a **multi-value column of its element type**,
written by the standard MV creators. `writeColumnIndexes` was already generic
over `IndexCreator`, which has a multi-value `add`, so this needed the shape
and the per-element dictIds rather than a new creator.
Shape follows the same rule the type already does, and for the same reason —
a column cannot be reshaped underneath documents that already wrote to it:
- **A declaration decides it, both ways.** A key declared single-value stays
single-value even when its values are collections, because that is what the
user asked for.
- **Otherwise the first value decides, and sticks.** A collection arriving
later on a scalar-shaped key is handled like any other value the column cannot
represent — stringified on a STRING key, a coercion failure on a typed one.
- **Elements that disagree resolve to STRING**, the same answer a key whose
values drift across rows gets, so a mixed array keeps every element rather than
losing the odd ones.
Edge cases, all covered by tests:
| case | result |
|---|---|
| scalar on a multi-value key | one element |
| absent document | one default element, as the standard creator writes for
an absent MV field |
| empty list | not a value — no elements, no type to infer, no empty state
on disk — so the key is absent for that document, exactly as a null value is |
| list longer than the mutable column's maximum | dropped and metered as a
coercion failure, rather than failing the segment |
### Both tiers
`MutableKeyColumn` gains a multi-value mode over
`FixedByteMVMutableForwardIndex`, so the consuming tier resolves the same
values by its own route. `OpenStructConsumingSealedParityTest` covers a
multi-value key element for element either side of `seal()`, pinning the values
themselves rather than only cross-tier equality.
### Tests
- `OpenStructMultiValueKeyTest` — element type, numeric elements,
disagreeing elements, declared SV with list values, declared MV with scalar
values, scalar on an MV key, shape stickiness, absent docs, empty lists.
- `OpenStructConsumingSealedParityTest` — new multi-value parity case.
78 OPEN_STRUCT tests across `pinot-spi` and `pinot-segment-local` pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
