xiangfu0 opened a new pull request, #19179: URL: https://github.com/apache/pinot/pull/19179
## Summary Adds five built-in virtual columns that expose common segment metadata to queries, alongside the existing `$docId` / `$hostName` / `$segmentName` / `$partitionId`. | Column | Type | Value | |---|---|---| | `$creationTime` | LONG | segment creation time, epoch millis | | `$startTimeMs` | LONG | segment time-range start, normalized to epoch millis | | `$endTimeMs` | LONG | segment time-range end, epoch millis | | `$totalDocs` | INT | documents stored in the segment (documents indexed so far while CONSUMING) | | `$segmentCrc` | STRING | segment CRC | ```sql SELECT $segmentName, $segmentCrc, $totalDocs, $creationTime FROM myTable GROUP BY 1, 2, 3, 4 ``` This makes a number of operational questions answerable directly in SQL — spotting replicas of a segment whose CRC has diverged, finding segments created before a rollout, or looking at per-segment document skew — without walking the segment metadata REST API. Like the existing built-ins, these are excluded from `SELECT *` and only materialize when named explicitly. ## Behavior Each column is a constant single-value column within a segment. Values are read from `SegmentMetadata` every time the column is built rather than baked into the field spec, so mutable segments — which rebuild their virtual data sources on every access — always observe current metadata. **Metadata that genuinely does not exist yet reads as SQL `NULL`, not a sentinel.** A CONSUMING segment has no time range and no CRC until it is committed, and a table without a time column never has a time range. This required teaching the virtual column path to carry a null value vector: `VirtualColumnIndexContainer` now serves `StandardIndexes.nullValueVector()`, and `VirtualColumnProvider` gained a `buildNullValueVector` hook that defaults to "no nulls", so existing providers are unaffected. Without it the placeholder in the forward index would be indistinguishable from a real value once null handling is enabled — e.g. `MIN($startTimeMs)` returning `Long.MIN_VALUE` on a consuming segment, and `IS NOT NULL` matching every row. `$totalDocs` counts the documents physically stored in the segment, so for an upsert table it also includes documents that have been replaced and are no longer returned by queries. ## Naming Only the time-range columns carry the `Ms` suffix. `SegmentMetadata#getStartTime()`/`#getEndTime()` return values in the time column's own unit, so unsuffixed `$startTime`/`$endTime` would be ambiguous about what they return; a creation time is always epoch millis throughout Pinot, so `$creationTime` is not. The rationale is recorded next to the constants. These strings are a permanent SQL contract, so this is worth a look during review. ## Supporting changes - **New `BuiltInVirtualColumns` in `pinot-spi`** is the single source of each column's name, data type, and single-value/multi-value shape. The broker side (`TableCache#addBuiltInVirtualColumns`) and the server side (`VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema`) now both build their field specs from it, so the two can no longer disagree on a type. Previously these specs were declared twice by hand, and a divergence would mean the broker declaring one type while the server produced another. - **`DefaultNullValueVirtualColumnProvider`** grew an overridable `getValue(context)` so it can back any per-segment constant column, plus a type check that names the offending column and provider instead of throwing a bare `ClassCastException` from inside segment loading. Behavior is unchanged for existing callers. - **Fixes a pre-existing bug in `SchemaInfo`**, which computed `getDimensionFieldSpecs().size() - 3` with a comment naming three virtual columns. That has been off by one since `$partitionId` was added, and these five columns would have made `GET /schemas/info` over-report user dimension counts by six in the controller UI. It now excludes built-in virtual columns by name, with a test pinning the invariant against the real `addToSchema` path. ## Cost Each column adds a constant dictionary, readers, and column metadata per immutable segment — roughly 1–2 KB per segment across the five, so on the order of 35 MB on a server holding 20k segments. There is no config gate, matching the existing four built-ins. ## Testing - New `SegmentMetadataVirtualColumnProviderTest` — values, column metadata, and the NULL path for both a missing `SegmentMetadata` and a real consuming-segment `SegmentMetadataImpl`, across the several representations of an unset creation time. - `LoaderTest` now iterates the full built-in set and asserts both the value path and the NULL path on a really-loaded segment. - `MutableSegmentImplTest` gained a test pinning the CONSUMING-segment shape (NULL time range/CRC/creation time, `$totalDocs` tracking documents indexed). - `OfflineClusterIntegrationTest.testSegmentMetadataVirtualColumns` covers both query engines: data schema, per-segment `$totalDocs` summing to the table row count, millisecond normalization checked against `DaysSinceEpoch * 86_400_000`, and filters on `$creationTime` and `$segmentCrc`. - Full `clean test` across `pinot-spi` → `pinot-common` → `pinot-segment-local` → `pinot-core` → `pinot-query-planner`, plus the integration tests above. All green. ## Docs The virtual-columns page in `pinot-docs` needs a matching update for these five columns and the NULL-on-unavailable behavior. ## Labels `feature`, `release-notes` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
