xiangfu0 opened a new pull request, #20035:
URL: https://github.com/apache/pinot/pull/20035
## Description
Timestamp-index override hints replace `DATETRUNC(...)` in the query
expression tree after aggregation functions have already captured their input
expressions. For example, `DISTINCTCOUNT(DATETRUNC('DAY', ts, 'MILLISECONDS'))`
continues projecting raw `ts` even when `$ts$DAY` is available.
Refresh compiled aggregation inputs, filtered-aggregation lookups, and
column dependencies after the rewrite. Retain the original aggregation slots so
duplicate SELECT aggregates and converging HAVING/ORDER-BY expressions preserve
the server/broker layout. Rewrite once before fetch planning, using derived
columns available on every selected segment; partially indexed segment sets
retain the original expression.
The original SQL can use the materialized DAY index after the existing
timestamp index is configured and built. This change does not create the index
automatically or add a new query option.
## Validation
- `./mvnw -ntp -pl pinot-core -am
-Dtest=QueryOverrideWithHintsTest,InstancePlanMakerImplV2Test
-Dsurefire.failIfNoSpecifiedTests=false -Dnet.bytebuddy.experimental=true
test`: **28 tests passed**, no failures/errors/skips.
- Regressions cover executable aggregate inputs, filtered aggregates with
null handling on/off, refreshed column dependencies and lookup keys, repeated
planning, preserved aggregation slots and canonical broker schema, missing
derived columns, and mixed-index segments in ordinary/streaming planning.
- Scoped Spotless, license formatting, Checkstyle, and license validation
passed for `pinot-core`.
- Independent source review found no actionable issues.
### Original-SQL local query check
The local external-Parquet reproduction executed the unchanged expression:
```sql
DISTINCTCOUNT(DATETRUNC('DAY', exportTimestamp, 'MILLISECONDS')) AS
active_days
```
The table had the existing DAY timestamp index configured and built. The
harness installs the native broker timestamp-override hints, then executes the
production planner, real JSON/range/inverted/null indexes, materialized DAY
column, two in-process server instances, serialization, and broker reduction.
It used this PR's compiled OSS classes plus a frozen external-Parquet extension
runtime that already skips unnecessary prefetch for JSON-index-only columns. No
query or planner compatibility shims were needed.
Fixture: 65,536 rows, two Parquet segments, 16 extra string leaves per
struct, 48-character strings, 10 warmups, 100 full-query measurements, and 100
filtered-count controls.
| Check | Before | After |
|---|---|---|
| Original SQL with DAY index and fail-on-Parquet-access guards | Fails on
timestamp `prefetchAllAsync` | Passes |
| Full-query results | Previously validated reference | All 100 match
complete 165-group values, exact distinct counts, names, types, and canonical
hash |
| Filtered-count controls | Independent expected count | All 100 match |
| Guarded Parquet forward value reads / context allocations | Guard rejects
unwanted access | 0 / 0 |
| Raw and decoded Parquet page caches, cleared before first query | — |
Remain empty; complete statistics unchanged after warmups and measurements |
The before control compiles the unmodified versions of the two production
files from the PR base; the after run loads the exact classes built from this
diff. Runtime inputs were fingerprinted and unchanged throughout each run.
JFR corroboration recorded during the check, across the 200 query windows:
173 CPU samples, zero samples with Parquet-forward-reader or prefetch frames,
and zero `DataLoss` events. Zero observed samples alone is not an I/O proof;
the reader guards and unchanged cold caches provide the access check.
This is a local post-setup query-path check, not a claim of zero
segment-load/footer/index-build or remote-index I/O, customer latency, or
concurrent p95. Diagnostic sources, generated fixtures, and profiles are kept
out of this production PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
