raghavyadav01 opened a new pull request, #19606:
URL: https://github.com/apache/pinot/pull/19606
OPEN_STRUCT keys a document one level deep, so a value inside a nested
object has no key naming it. `{"device": {"os": "ios"}}` has exactly one key,
`device`, whose value is an object — no column is ever materialized for `os`,
no predicate can reach it, and every query that wants it decodes the object per
row.
### What this adds
`maxNestedKeyDepth` on the open_struct index config, default `1` (today's
behavior, unchanged). Above 1, **the path becomes the key**:
```
{"device": {"os": "ios", "ver": 17}} maxNestedKeyDepth = 2
device -> {"os":"ios","ver":17} (container, JSON text)
device.os -> "ios" (own column, own type, own index)
device.ver -> 17
```
`device.os` is then materialized, indexed, filtered and projected like any
other key. Nothing about the key/value contract changes: `.` is an ordinary
character in a key, so `col['device.os']` is already valid syntax, and a dotted
column name is already supported on disk — `ColumnIndexUtils.parseIndexMapKeys`
scans index-map keys backwards for exactly this reason.
### Semantics
- **The container keeps its own entry**, serialized as JSON text, so
`col['device']` still returns the whole object after its leaves are split out.
- **Containers are held out of automatic dense selection.** Their leaves are
separate keys already competing for the same budget, and a column of JSON blobs
is not what dense materialization is for. `denseKeys` still pins one by name.
- **Depth counts path segments** and is the only bound: `2` reaches `a.b`,
`3` reaches `a.b.c`. An object sitting at the limit is emitted as JSON text
rather than dropped, so a limit that is too low costs addressability, not data.
- **Lists stay single values.** Their elements have no names to build a path
from.
- Opt-in per column, because depth bounds recursion but not breadth — a wide
object contributes a key per entry.
### Parity
Both the sealed build path (`OpenStructColumnSplitter`) and the consuming
path (`MutableOpenStructIndex`) flatten through the same helper, so a nested
value resolves under the same key either side of the seal boundary.
`OpenStructConsumingSealedParityTest` covers the leaf and the container, and
pins the values themselves rather than only cross-tier equality.
### Tests
- `OpenStructKeyFlattenerTest` — depth, containers, lists, empty objects,
null keys/values, a literal dot in a key.
- `OpenStructNestedKeyTest` — default depth unchanged, leaf gets its own
typed column, container stays readable, configured dense container, declared
child spec on a path key, object below the depth limit not lost, sparse leaf
round-trips through the blob.
- `OpenStructConsumingSealedParityTest` — new nested-path-key case.
All 68 existing OPEN_STRUCT tests in `pinot-segment-local` and 30 in
`pinot-core` still pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
