tobegit3hub commented on issue #10225: [SPARK-12196][Core] Store/retrieve blocks from different speed storage devices by hierarchical way URL: https://github.com/apache/spark/pull/10225#issuecomment-478921982 Thanks @yucai . I don't think it is as good as your hierarchical proposal. We want to use one SSD for shuffle and cache for performance and degrade to use HDD if no enough space. One SSD may cover mostly scenarios so we think the hierarchical way is the right way to go on. BTW, can use only SSD for shuffle and HDD for cache? Now we submit Spark jobs to Yarn and it relies on the configuration of `yarn.nodemanager.local-dirs` which has to specified HDD or SSD.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
