tarun11Mavani opened a new pull request, #19154:
URL: https://github.com/apache/pinot/pull/19154
Virtual SparseKeyDataSource over the sparse blob column makes unmaterialized
keys readable through the standard per-key DataSource seam. A sparseKeys
manifest in parent column metadata short-circuits truly absent keys without
touching the blob. An opt-in JSON index on the sparse column accelerates
eligible STRING-typed EQ/IN/NOT_EQ/NOT_IN filters.
Components:
- OpenStructSparseBlobReader: shared per-parent blob parser with ThreadLocal
LRU cache (10,000 entries, one projection block)
- SparseKeyDataSource: virtual forward reader + lazy null vector per key
- ImmutableOpenStructDataSource routing: materialized -> sparse virtual ->
manifest-absent null
- MapFilterOperator 4-tier dispatch: materialized index -> sparse JSON-index
fast path -> virtual scan -> expression fallback
- OpenStructIndexConfig.sparseJsonIndex (default false): opt-in JSON index
on the sparse blob column
- JSON-index fast path refusal rules: non-STRING type, default-value
predicate, negation with null-handling-on
Back-compat: segments without the manifest fall back to virtual scan for
every unmaterialized key (correct but unoptimized).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
