xuechendi opened a new pull request #24322: [SPARK-27412][Shuffle]Add a new shuffle manager: PmemShuffleManager URL: https://github.com/apache/spark/pull/24322 PmemShuffleManager is used to write/read shuffle data and external sorter spilled data to Persistent Memory Device, and leverage PMDK to fully drive PMem device and retrieve exellent bandwidth and latency. Signed-off-by: Chendi Xue <[email protected]> ## What changes were proposed in this pull request? No codes change in current spark, while add a new shuffle manager with a bunch of new files. ## How was this patch tested? This patch passed functional test, running spark on yarn with hdfs as back-end and performed terasort, repartition and random forest tests. Please review http://spark.apache.org/contributing.html before opening a pull request.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
