Probability of a complete node failure is low. I would rely on data lineage and accept the reprocessing overhead. Another option would be to Write on distributed FS but it will drastically reduce all your jobs speed
Le 20 déc. 2017 11:23, "chopinxb" <chopi...@gmail.com> a écrit : > Yes，shuffle service was already started in each NodeManager. What i mean > about node fails is the machine is down，all the service include nodemanager > process in this machine is down. So in this case, shuffle service is no > longer helpfull > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
