zhuqi-lucas commented on PR #24445: URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5351275370
On the perf question: `physical_sorted_union_order_by_50_uint64` is the shape this touches, and the bot has it at ~10% across three runs on different commits: ``` 82edbc3b 404.4ms -> 368.9ms 1.10 09d9cd2d 416.0ms -> 369.8ms 1.13 95cc5360 410.3ms -> 373.9ms 1.10 ``` The smaller variants move less, which is what the mechanism predicts: more columns in the sort order means a larger equivalence group to re-project on each rebuild. The query is not public and I could not reduce it to an MRE, but I should also be straight about why its number was so much larger. That codebase predates `EnsureRequirements` and still runs `EnforceSorting` six times plus `EnforceDistribution` twice, so the projection chain is rebuilt eight times per plan rather than once. The saving is per rebuild, so consolidating those passes -- which upstream already did -- caps most of it. Quoting 45% here without that context was misleading and I will rewrite the description around the numbers above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
