Hi, I am trying to load the Graphx example dataset(LiveJournal, 1.08GB) through Scala Shell on my standalone multicore machine(8 cpus, 16GB mem), but an OutOfMemory error was returned when below code was running,
val graph = GraphLoader.edgeListFile(sc, path, minEdgePartitions = 16).partitionBy(PartitionStrategy.RandomVertexCut) I guess I should set some parameters to JVM? like "-Xmx5120m" But how to do this in Scala Shell? I directly used the "bin/spark-shell" to start spark and seems everything works correctly in WebUI. Or, I should do parameters setting at somewhere(spark-1.0.1)? Best, Yifan LI
