I have a simple job , that reads data => union => filter => map and the count
1 Job started 2402 tasks read 149G of input. I started the job with different number of executors 1) 1 --> 8.3 mins 2) 2 --> 5.6 mins 3) 3 --> 3.1 mins 1) Why is increasing the cores speading up this app ? 2) I started the job with --num-executors 9973 but when i click executors tab i see 330 executors. So can i start the job with --num-executors 330 as i get only that from YARN cluster ? 3) I had set the split size to 64 MB but when i start the job with --executor-memory 14g , how do i decide how much memory i need ? also as the cores are increasing how do i get that into the calculations ? 4) as the speed is getting better how far can i go with increasing executors ? -- Deepak
