I am facing a somewhat confusing problem:
My spark app reads data from a database, calculates certain values and then
runs a shortest path Pregel operation on them. If I save the RDD to disk and
then read the information out again, my app runs between 30-50% faster than
keeping it in memory, plus the in memory solution crashes with GC overhead
exceptions every other time and I get
"TaskSetManager: Stage 12 contains a task of very large size (890 KB). The
maximum recommended task size is 100 KB" errors on the memory solution.
Here is the part:
Slower:
val dailyRate:RDD[OneCalc] = getDaily(originalData)
val distance: RDD[Edge[Int]] = dailyRate.map(x => Edge(x.fromEdge, x.toEdge,
x.avg))
val graph = Graph(datapoints, distance, MySpark.defaultRoute,
StorageLevel.MEMORY_AND_DISK_SER, StorageLevel.MEMORY_AND_DISK_SER)
=> Pregel
Faster:
val dailyRate:RDD[OneCalc] = getDaily(originalData).collect()
sc.parallelize(dailyRate.map(x=>s"${x.from},${x.to},${x.avg}"))
.repartition(1).saveAsTextFile("file:///tmp/spark")
val distance = sc.textFile("file:///tmp/spark/part-00000").flatMap(x=> {
val line = x.split(",")
OneCalc(line)
}).map(x => Edge(x.fromEdge, x.toEdge, x.avg))
val graph = Graph(datapoints, distance, MySpark.defaultRoute,
StorageLevel.MEMORY_AND_DISK_SER, StorageLevel.MEMORY_AND_DISK_SER)
=> Pregel
Any idea why this is happening?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Writing-and-reading-file-faster-than-memory-option-tp20588.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]