A CSV element like "3.2," takes 4 bytes as text on disk, but, as a Double will always take 8 bytes. Is your input like this? that could explain it.
You can map to Float in this case to halve the memory, if that works for your use case. This is just kind of how Strings and floating-point work in the JVM, nothing Spark-specific. On Mon, Oct 13, 2014 at 9:12 PM, Aris <[email protected]> wrote: > Hi guys, > > I am trying just parse out values from a CSV, everything is a numeric > (Double) value, and the input text CSV data is about 1.3 GB in size. > > When inspect the Java Heap space used by SparkSubmit using JVisualiser VM, I > end up eating up 8GB of memory! Moreover, by inspecting the BlockManager > input, the original 1.3 GB input CSV file goes into an RDD[Array[Double]], > which is now about 3.6 GB in size from the BlockManager debug output. > > This means just parsing the CSV of doubles into a very primitive container > format for Spark almost triples the size of the input... what is going on > here? It gets worse because the TOTAL heap space taken up is over 8GB as I > mentioned. > > From trying to work through Java Heap output and inspecting it with Eclipse > Memory Analyze and Jvisualizer VM, it looks like the parsed-out Double > values end up taking up twice the amount of memory...meaning about 7.2 GB of > memory, which explains most of the Java Heap space. > > My code is very simple, does anybody know why I am eating up so much memory? > > val input: RDD[Array[Double]] = sc.textFile(inFile).map(strLine => > strLine.split(",").map(_.toDouble) > ) cache() > val grouped: RDD[(Double, Iterable[Array[Double]])] = > input.groupBy(_.last) > > > Thank you! > Aris --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
