comphead commented on issue #26541: URL: https://github.com/apache/datafusion/issues/26541#issuecomment-6884880371
Another data point, from Comet, which runs Spark's window functions in `BoundedWindowAggExec`. One stage sorts and windows about 1.07B rows in 216 tasks of about 5M rows: - `last_value(s) IGNORE NULLS OVER (PARTITION BY k1, k2, k3 ORDER BY ts RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)`, where `s` is a struct of 5 fields, with about 2 rows per partition - `max(i) OVER (PARTITION BY k1, k3 ORDER BY ts)` with about 5 rows per partition | Operator | `elapsed_compute`, all tasks | Per task, median | Per partition (estimate) | |---|---|---|---| | `last_value` window | 7,768 s | 36.0 s | about 15 µs | | `max` window | 1,908 s | 8.8 s | about 10 µs | Spark's whole stage, including its sorts, windows and aggregates, takes 7,275 s of CPU. The partition counts are estimated from the data generator, not measured. The struct argument makes the per-partition cost higher, because each one-row partition goes through a `ScalarValue::Struct` and a one-row `StructArray`, and the per-partition outputs are then concatenated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
