2010YOUY01 commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5552190361
> > 3. Ordering: we stop advertising it — no re-sort. And DataFusion is the outlier for advertising it at all > > Background, since this corner of DataFusion is niche: operators declare to the planner which input orderings survive them (maintains_input_order) > > I am not sure I would call the sort preserving features niche -- there are many users who use it heavily in my experience (including us at InfluxData). One thing we could to do reduce the impact would be to extend the analysis that [@zhuqi-lucas](https://github.com/zhuqi-lucas) has done (in Sort Pushdown) to determine when the advertised sort order was actually used. > > (I just got your images, I am working through them now) I also found that this order-preserving property can be a blocker for this issue while I was prototyping the HJ refactor. I'm leaning towards temporarily removing it. The reason is that the HJ memory-limited fallback is a workload-agnostic feature, while the ordering optimization is workload-specific. Giving up an optimization that targets certain workloads in favor of a more generally applicable feature seems like a good architectural trade-off to me. If we give up that ordering property, the physical implementation of HJ can also be simplified a bit now. However, I believe we want to bring back those optimizations in the long term. The main workloads affected by the output ordering are: (covered by existing tests) ``` # type 1: top K after HJ TopK (expr = r.c1, limit =10) -- HashExec (right order preserving) # type 2: aggregate after HJ AggregateExec (group_by = r.c1) -- HashExec (right order preserving) ``` Type 1 can be optimized in a similar way with a dynamic filter, without adding too much extra complexity inside the operator. To implement type 2, we might need to maintain two parallel HJ implementations (order-preserving vs. non-order-preserving) plus extra optimizer rules, which is a lower ROI for the engineering effort involved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
