yuelimv commented on issue #23707: [SPARK-26793][CORE]Remove spark.shuffle.manager URL: https://github.com/apache/spark/pull/23707#issuecomment-459543790 @10110346 the option `spark.shuffle.manager` allows users to easily plug in other shuffle manager implementation, and I personally think it is a good option to preserve. As @felixcheung mentioned, we are currently having many on-going discussions and developments related to more reliable and flexible shuffle (for details, see [SPARK-25299]( https://issues.apache.org/jira/browse/SPARK-25299)) For instance, one effort made by us, named [Splash](https://github.com/MemVerge/splash), allows the use of other storage backends and network transports for exchanging shuffle data. Splash is configured via setting the `spark.shuffle.manager` option (see its README for more details).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
