Curious how SPARK-25299 (where file tracking is pushed to spark drivers, at
least in option-5) interacts with Splash. The shuffle data location in
SPARK-25299 would now have additional "fallback" logic for recovering from
executor loss.
On Thu, Jan 3, 2019 at 6:24 AM Peter Rudenko
wrote:
> Hi
Hi Matt, i'm a developer of SparkRDMA shuffle manager:
https://github.com/Mellanox/SparkRDMA
Thanks for your effort on improving Spark Shuffle API. We are very
interested in participating in this. Have for now several comments:
1. Went through these 4 documents:
Hi everyone,
Earlier this year, we proposed SPARK-25299, proposing the idea of using other
storage systems for persisting shuffle files. Since that time, we have been
continuing to work on prototypes for this project. In the interest of
increasing transparency into our work, we have created
