Hi Spark Users, Uber open sourced Remote Shuffle Service ( https://github.com/uber/RemoteShuffleService ) recently. It works with open source Spark version without code change needed, and could store shuffle data on separate machines other than Spark executors.
Anyone interested to try? Also we are still actively developing. Please let us know if you are interested to work together on it as well :) Best, Bo