I am confused about the process local and node local, too. In my current understanding of Spark, one application typically only has one executor in one node. However, node local means your data is in the same host, but in a different executor.
This further means node local is the same with process local unless one node has two executors, which could only happen when one node has two Workers. Waiting for further discussion .. On Mon, Apr 14, 2014 at 10:13 AM, Nathan Kronenfeld < [email protected]> wrote: > I've a fairly large job (5E9 records, ~1600 partitions).wherein on a given > stage, it looks like for the first half of the tasks, everything runs in > process_local mode in ~10s/partition. Then, from halfway through, > everything starts running in node_local mode, and takes 10x as long or more. > > I read somewhere that the difference between the two had to do with the > data being local to the running jvm, or another jvm on the same machine. > If that's the case, shouldn't the distribution of the two modes be more > random? If not, what exactly is the difference between the two modes? > Given how much longer it takes in node_local mode, it seems like the whole > thing would probably run much faster just by waiting for the right jvm to > be free. Is there any way of forcing this? > > > Thanks, > -Nathan > > > -- > Nathan Kronenfeld > Senior Visualization Developer > Oculus Info Inc > 2 Berkeley Street, Suite 600, > Toronto, Ontario M5A 4J5 > Phone: +1-416-203-3003 x 238 > Email: [email protected] > -- Dachuan Huang Cellphone: 614-390-7234 2015 Neil Avenue Ohio State University Columbus, Ohio U.S.A. 43210
