Hi Aaron, > Clearly either of the latter two solutions [no compression or 1 executor/node] > will produce a significant slowdown
Just curious, but why would turning off compression lead to a significant slow down? Just more IO, I guess? FWIW, the job we'd been discussing with 18k partitions, I tried with a 512mb goal partition size, which led to 2k partitions, and it completed just fine. Unfortunately, I tried the 512mb goal partition size with one of our production jobs, and it blew up with an OOME in the AppendOnlyMap reducer side of things, which I'm pretty sure means the partition was now too big to have 2 of them fit in memory. I don't have a heap dump for that one, but could get one fairly easily. I'll discuss this with a few guys here; thinking we'll either try no compression, or move to just using m1.xlarges. > I am currently investigating shuffle file performance, and thanks to > your feedback here, I'll additionally investigate the memory > overheads inherent in shuffling as well. Cool. Sounds great. Thanks for all the help, I appreciate it. - Stephen
