zwangsheng commented on PR #2192: URL: https://github.com/apache/incubator-celeborn/pull/2192#issuecomment-1873571331
> In general, let us move away from making changes to spark (or any other supported compute platform) to use Celeborn. If there is required, missing functionality - we should work with those communities to add required support: there might be reasons why it does not make sense to do so, and the relevant community is the best judge to evaluate it @mridulm Thank you for noticing this feature. When designing this function, we tried to reduce the modification of Spark. The Shuffle Manager serves as an adaptable plug-in for Spark. Presently, the Spark framework exclusively supports the shuffle handler and corresponding shuffle writer/reader for extension implementation. However, it lacks an interface for the Preferred Location crucial for scheduling processes. It is both rational and pivotal for the Spark framework to expose essential methods to plug-ins without altering its own logic. The ideal scenario would involve persuading the upstream community (Spark) to embrace the minor adjustments required. At present, we are doing some functional and performance verification of this PR. If there is a good improvement, it will be listed in detail in PR and cooperate with the upstream community. Finally, thanks for your attention! :) The changes to Spark are small and reasonable. Also FYI @waitinfuture -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
