Hi everyone,
I am very new to Spark, so as a learning exercise I've set up a small
cluster consisting of 4 EC2 m1.large instances (1 master, 3 slaves), which
I'm hoping to use to calculate ngram frequencies from text files of various
sizes (I'm not doing anything with them; I just thought this would be
slightly more interesting than the usual 'word count' example). Currently,
I'm trying to work with a 1GB text file, but running into memory issues.
I'm wondering what parameters I should be setting (in spark-env.sh) in
order to properly utilize the cluster. Right now, I'd be happy just to
have the process complete successfully with the 1 gig file, so I'd really
appreciate any suggestions you all might have.
Here's a summary of the code I'm running through the spark shell on the
master:
def ngrams(s: String, n: Int = 3): Seq[String] = {
(s.split("\\s+").sliding(n)).filter(_.length == n).map(_.mkString("
")).map(_.trim).toList
}
val text = sc.textFile("s3n://my-bucket/my-1gb-text-file")
val mapped = text.filter(_.trim.length > 0).flatMap(ngrams(_, 3))
So far so good; the problems come during the reduce phase. With small
files, I was able to issue the following to calculate the most frequently
occurring trigram:
val topNgram = (mapped countByValue) reduce((a:(String, Long), b:(String,
Long)) => if (a._2 > b._2) a else b)
With the 1 gig file, though, I've been running into OutOfMemory errors, so
I decided to split the reduction to several steps, starting with simply
issuing countByValue of my "mapped" RDD, but I have yet to get it to
complete successfully.
SPARK_MEM is currently set to 6154m. I also bumped up the
spark.akka.framesize setting to 500 (though at this point, I was grasping
at straws; I'm not sure what a "proper" value would be). What properties
should I be setting for a job of this size on a cluster of 3 m1.large
slaves? (The cluster was initially configured using the spark-ec2 scripts).
Also, programmatically, what should I be doing differently? (For example,
should I be setting the minimum number of splits when reading the text
file? If so, what would be a good default?).
I apologize for what I'm sure are very naive questions. I think Spark is a
fantastic project and have enjoyed working with it, but I'm still very much
a newbie and would appreciate any help you all can provide (as well as any
'rules-of-thumb' or best practices I should be following).
Thanks,
Tim Perrigo
