Jackie-Jiang opened a new pull request, #20074: URL: https://github.com/apache/pinot/pull/20074
## Summary Follow-up to #20063. When the multi-stage leaf returns final results (the `is_leaf_return_final_result` hint, or a colocated DIRECT aggregate with `serverReturnFinalResult`), some aggregations fail: - Array functions (`ARRAYAGG`, `SUMARRAYLONG` / `SUMARRAYDOUBLE`, `HISTOGRAM`, the funnel functions) return fastutil lists such as `IntArrayList`, while their columns hold arrays. The final stage casts the transported arrays to `Comparable` before merging and throws `ClassCastException`, and a leaf executed by the single-stage engine leaves the lists in array columns, which fails when the block is serialized. - Raw sketch functions (`DISTINCTCOUNTRAWHLL` etc.) put their result object into a `STRING` column, which also fails when the block is serialized. - An MSE sort on an array column throws because `SortUtils` casts values to `Comparable`. The root cause is that a final result differs from the stored value of its column type, so every boundary has to convert between the two. This PR removes that difference: - The final result of an `AggregationFunction` is now the stored value of `getFinalResultColumnType()`, e.g. `int[]` for `INT_ARRAY`, so it is transported, sorted and merged without conversion. The `FinalResult extends Comparable` bound is removed, along with the list/array conversions in `DataSchema`, `TypeUtils`, `DataTableBuilderUtils`, `AggregationResultsBlock`, `AggregationFunctionUtils` and `GroupByDataTableReducer`, and the `ArrayListUtils` methods they used. - The new `StoredValueComparator` in `pinot-spi` orders stored values: `Comparable` values by their natural order, arrays element by element. `TableResizer`, `SortUtils` and the MSE group trim use it. - A `STRING` final result can still be an object stored as its `toString()` (raw sketches, `SUMPRECISION`), so the MSE leaf converts final results to the stored types of the plan. ## Compatibility - The wire format, MSE plans and query responses do not change. - The erasure of `extractFinalResult` and `mergeFinalResult` changes from `Comparable` to `Object`, so `AggregationFunction` implementations compiled against the old interface must be recompiled. An implementation with an array final type must return an array rather than a fastutil list. ## Follow-up A TODO on `AggregationFunction#extractFinalResult` lists what is left to make final results behave the same in SSE and MSE. Each item changes types on the wire or in MSE plans, so it needs a rollout gate: - Raw sketch final results order by the sketch estimate where the object is compared, but by the serialized string after transport. - `SUMPRECISION` returns `STRING` in SSE but `BIG_DECIMAL` in MSE, so SSE cannot merge its final results from several servers. - `OBJECT` final results have no serialization of their own. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
