cyb70289 commented on pull request #9435: URL: https://github.com/apache/arrow/pull/9435#issuecomment-783843292
Other thoughts about O(1) space quantile kernel performance: TDigest is good at estimate skewed quantiles, like 0.01, 0.99. For normal quantiles between 0.1 ~ 0.9, the old school online quantile algorithm P-Square [1] is good enough, it should be faster than TDigest. Maybe we can add a p-square kernel for additional cases, or auto select tdigest/p-square per input options. [1] https://www.cse.wustl.edu/~jain/papers/ftp/psqr.pdf ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
