anuragrai16 opened a new pull request, #18725:
URL: https://github.com/apache/pinot/pull/18725
## Summary
MSE Lite silently truncates query results when a query runs without an
explicit `LIMIT` clause — the planner inserts a
`PhysicalSort(FETCH=liteModeLimit)` at the leaf-stage boundary, but returns
incomplete results with no user-facing signal. This PR surfaces that truncation
via three new fields in the broker response.
## Changes
**Planning side** (`pinot-query-planner`):
- `PhysicalPlannerContext`: two new fields to track when an implicit sort
was injected and the effective limit
- `LiteModeSortInsertRule`: sets the context flags in all three
implicit-limit branches (sort-with-fetch, aggregate-no-limit,
new-sort-insertion)
- `QueryEnvironment.QueryPlannerResult`: exposes the planner context flags
to the broker
**Transport** (`pinot-spi`, `pinot-common`):
- New query option key `LITE_MODE_IMPLICIT_LEAF_STAGE_LIMIT` — injected by
the broker after planning so servers can detect truncation
- New `DataTable.MetadataKey.LITE_MODE_LEAF_STAGE_LIMIT_REACHED` (id 44) —
set by the server when the implicit limit is binding
- `QueryOptionsUtils.getLiteModeImplicitLeafStageLimit()` — parser for the
new query option
**Server-side detection** (`pinot-core`):
- `InstanceResponseOperator.buildInstanceResponseBlock()`: checks `numRows
>= implicitLimit` and sets the metadata flag (handles aggregation queries via
non-streaming combine path)
- `StreamingInstanceResponseOperator.getNextBlock()`: tracks
`totalRowsStreamed` during the streaming loop and sets the metadata flag after
completion (handles selection queries — the common case for MSE Lite)
**Stats propagation** (`pinot-query-runtime`):
- `LeafOperator.StatKey.LITE_MODE_LEAF_STAGE_LIMIT_REACHED` + `case` in
`mergeExecutionStats()` switch — mandatory, as `default` throws
`IllegalArgumentException`
- Auto-maps to `BrokerResponseNativeV2.StatKey` by enum constant name
**Broker response** (`pinot-common`, `pinot-broker`):
- `BrokerResponseNativeV2`: new
`StatKey.LITE_MODE_LEAF_STAGE_LIMIT_REACHED`, accessor/merge methods, three new
JSON fields in `@JsonPropertyOrder`
- `isPartialResult()` updated to include `isMseLiteLeafStageLimitReached()`
- `MultiStageBrokerRequestHandler`: injects query option before dispatch;
sets planning-time response fields (`effectiveLimit`,
`fanOutAdjustedLimitApplied`) and logs a WARN after execution
## New response fields
| Field | Type | When present |
|---|---|---|
| `mseLiteLeafStageLimitReached` | boolean | Always (false if not reached) |
| `mseLiteLeafStageEffectiveLimit` | int | Only when planner injected an
implicit sort |
| `mseLiteFanOutAdjustedLimitApplied` | boolean | Only when planner injected
an implicit sort |
## Example response (limit reached)
```json
{
"numRowsResultSet": 8,
"partialResult": true,
"mseLiteLeafStageLimitReached": true,
"mseLiteLeafStageEffectiveLimit": 2,
"mseLiteFanOutAdjustedLimitApplied": false
}
```
## Test plan
- Unit Tests
- Manual verification against STREAM quickstart with 17 test cases covering
both streaming (selection) and non-streaming (aggregation) operator paths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
