I have a flatmap that takes a line from an input file, splits it by tab into "words" then returns an array of tuples consisting of every combination of 2 words. I then go on to count the frequency of each combination across the whole file using a reduce by key (where the key is the tuple).
I am running this now and I am getting terrible performance. My guess is that Spark is trying to run 12 tasks in parallel since each machine has 12 cpus but the memory consumed by each task is far more than 1/12th of the memory on each machine. Metrics: 16 node cluster, 12 cpus each, 60GB allocated to spark each. Input file: 119.5 MB / 84134 lines After 3 days (3295.7 h across all tasks) - Shuffle write: 197.1 GB / 22701021927 Here are the metrics from the worst performing task (so far): Duration: 58.8 h GC time: 19.0 h Input Size / Records: 383.9 KB (hadoop) / 153 Write Time: 4 s Shuffle Write Size / Records: 2.4 GB / 283297353 Shuffle Spill (Memory): 30.4 GB Shuffle Spill (Disk): 1941.4 MB I used 400 partitions when reading the input file. I'm guessing this would be the first thing to change. Does anyone have any suggestions of what to increase it to? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-deal-with-an-explosive-flatmap-tp23277.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
