Dandandan commented on PR #15562: URL: https://github.com/apache/datafusion/pull/15562#issuecomment-2779629454
My current intuition about it is twofold: * It doesn't show up as clearly in a normal flamegraph as it doesn't show you the profile per thread but only the "overall" samples, so the percentage will be divided by the number of total samples (e.g. 2% instead of 20%) * SortPreservingMergeStream polls the required input streams - so whenever all the child streams don't produce any new batches it will keep on polling - therefore `self.uninitiated_partitions.clone()` comes up very often in sample as it is a relatively expensive part of the "polling" loop. Whenever there is useful work todo, `self.uninitiated_partitions.clone()` is not a bottleneck, processing the batches is much more expensive. Optimizing it will mainly move the CPU power to other parts of the polling loop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
