raghavyadav01 opened a new pull request, #19858: URL: https://github.com/apache/pinot/pull/19858
An `OPEN_STRUCT` column keeps its JSON index on the sparse blob child rather than on the column itself, so `dataSource.getJsonIndex()` returns `null` and `JSON_MATCH` is refused outright with `Cannot apply JSON_MATCH on column: <col> without json index` — even though the blob is a real JSON document and the index over it can answer paths into it. That left paths into arrays and nested values with no index-served spelling at all. `col['key']` reaches the same index through `MapFilterOperator`, but it can only name a whole key, so anything inside an array had to be pulled apart with `JSON_EXTRACT_SCALAR`, which scans. This makes `JSON_MATCH` fall back to the blob's index when the column has none of its own. ## The part that needs care The index may answer only what the blob actually holds. A materialized key is lifted into its own dense column and is **never written to the blob**, so a path naming one has no postings — serving it from the index would answer "no documents match" confidently and wrongly. `col['key']` is immune to this because `MapFilterOperator` resolves the key's own `DataSource` first and takes the index only when the key is sparse. `JSON_MATCH` names its paths inside an opaque filter string, so the same check has to be done at plan time: the filter is parsed and every top-level key it names is checked against `isMaterialized` and `isPathIndexed`. Any path that fails leaves the existing refusal in place. Refusing is the right failure here — `JSON_MATCH` has no scan to fall back to, so the alternative to "cannot apply" would be a confidently empty answer. ## Tests Five cases in `FilterPlanNodeTest`: - a sparse path is served from the blob's index - a materialized key is refused rather than answered from the blob, and the index is never consulted - a path the index does not hold is refused - an `AND` where only one of two paths is materialized is refused — every path has to clear the bar - a column with no blob index is refused exactly as before `FilterPlanNodeTest` 33/33, plus `MapFilterOperatorOpenStructTest`, `JsonMatchQueriesTest`, `JsonMatchQueriesWithDisableUnnestTest`, `ProjectionBlockOpenStructTest` and `JsonPathQueriesTest` green (51 tests). Verified end to end on a cluster as well: a path into an array element returns the same count as the equivalent `JSON_EXTRACT_SCALAR` predicate with `numEntriesScannedInFilter = 0`, and a path naming a materialized key still refuses. ## Behaviour change Only for `JSON_MATCH` on an `OPEN_STRUCT` column, which fails today. Every other column is untouched: the new branch is reached only when `getJsonIndex()` and the composite index lookup both return `null` **and** the data source is an `OpenStructDataSource`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
