yashmayya opened a new pull request, #19954: URL: https://github.com/apache/pinot/pull/19954
## Problem Since #19412, a window with `PARTITION BY x ORDER BY y` runs on one worker for the whole cluster, and `stageParallelism` has no effect. Grouped `WITHIN GROUP` aggregates have the same problem. - #19412 puts a `Sort` with no fetch above the hash exchange of these operators. - #17347 marks every `Sort` that reads from a mailbox as a final sort, and asks for a singleton worker. Together, they put the whole window stage on one worker. Example: on TPC-H SF10 with 2 servers, `ROW_NUMBER() OVER (PARTITION BY l_suppkey ORDER BY l_extendedprice DESC)` over `lineitem` (60M rows) ran on one worker. ## Fix `DispatchablePlanVisitor` asks for a singleton worker only for a real final sort: - the `Sort` has a fetch or an offset, or - the exchange below it is not hash-partitioned on keys (for example, the `hash[]` exchange of a top-level `ORDER BY`). A `Sort` with no fetch over a keyed hash exchange orders each partition only, so its stage keeps the normal worker assignment. The #17347 behavior for `ORDER BY` / `LIMIT` does not change. ## Tests - `QueryCompilationTest`: partitioned windows and grouped `WITHIN GROUP` aggregates get one worker per server (times `stageParallelism`). `ORDER BY`, `ORDER BY ... LIMIT`, `OVER (ORDER BY ...)` and ungrouped `WITHIN GROUP` still get one worker. A window under a final `ORDER BY` gets both. - New `DispatchablePlanVisitorTest`: each fetch / offset / distribution case. - `ExplainPhysicalPlans.json`: implementation plan of a partitioned window. The new worker-count tests fail without the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
