Weijun-H commented on PR #14411: URL: https://github.com/apache/datafusion/pull/14411#issuecomment-2658410453
> > > > I wonder why tpch_mem_sf10 is slower for some queries? Might it be possible the created memtable is not created evenly because of the new round robin (that might be fixable e.g. by introducing another repartition after memoryexec). > > > > > > > > > Thank you for the advice @Dandandan. We will certainly check that after completing on-demand optimizations. > > > > > > I agree with @berkaysynnada because `OnDemandRepartition` is not set by default; it should be enabled when necessary. However, it is a good point for the following optimization, @Dandandan 👍. > > I wonder if we can set it to default once we know it generally leads to faster execution? Yes, I believe it would be better to set `OnDemandRepartition` by default later, as it generally results in faster execution. For now, it is acceptable to disable it and wait for further optimization in iterations. > Specifically, I think we can try [this approach](https://github.com/apache/datafusion/pull/13707) together with on-demand repartition 🤔 It could be an optimization, I will try to rebase it later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
