Hi Donni, Please check spark dynamic allocation and external shuffle service .
On Fri, 27 Apr 2018 at 2:52 AM, Donni Khan <[email protected]> wrote: > Hi All, > > Is there any way to change the number of executors/cores during running > Saprk Job. > I have Spark Job containing two tasks: First task need many executors to > run fastly. the second task has many input and output opeartions and > shuffling, so it needs few executors, otherwise it taks loong time to > finish. > Does anyone knows if that possible in YARN? > > > Thank you. > Donni > -- *Regards,* *Jogesh Anand*
