avantgardnerio commented on PR #24035: URL: https://github.com/apache/datafusion/pull/24035#issuecomment-5170587834
@neilconway I'm trying to speed up window functions using parallel [prefix scans](https://www.cs.cmu.edu/~guyb/papers/Ble93.pdf). I am presently [incubating this](https://github.com/apache/datafusion-ballista/pull/2211) in Ballista, and this is the minimum API exposure that I need to do it for non-decomposable operations like `approx_distinct()` (vs others like AVG, STDDEV via Welford / Chan, etc). Though, the work is certainly not limited to Ballista, it has shown improvement in [DataFusion as well](https://github.com/coralogix/arrow-datafusion/pull/426). The jury is still out about re-partition cost vs performance benefit, but the signs are hopeful: <img width="1560" height="600" alt="image" src="https://github.com/user-attachments/assets/81a48c0a-86aa-427a-81eb-f00fdfa634e2"; /> And at least from a [big-O time perspective](https://www.vldb.org/pvldb/vol8/p1058-leis.pdf) (table 1) it should be optimal for some queries (`select my_agg() over unbounded preceding`...) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
