advancedxy commented on PR #1660: URL: https://github.com/apache/incubator-uniffle/pull/1660#issuecomment-2068558005
> > For this particular feature, I think MR/Tez would benefit from that substantially(to be frankly, I had developed a similar shuffle system back ~6-7 years ago). The spark integration is less exciting though. > > But if we could introduce the extra plan to optimize the sortMergeJoin from sortExec + exchange to uniffleSortedExchangeExec that will benefit from this feature. Is this possible? emmm, maybe in your own forked Spark. For open source Spark, it might be possible to inject a strategy rule via SparkSQLExtension.`injectQueryPostPlannerStrategyRule` to replace `SortExec`. However, when it comes to Spark's internal EnsureRequirements, you have to touch/modify the internal logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
