Thank you Sean. Moving over my data types from Double to Float was an
(obvious) big win, and I discovered one more good optimization from the
Tuning section --
I modified my original code to call .persist(MEMORY_ONLY_SER) from the
FIRST import of the data, and I pass in "--conf spark.rdd.compress=true"
from the spark-submit command line to do more intensive compression of my
RDDs; this helped keep my data size in check.
val input: RDD[Array[Float]] = sc.textFile(inFile).map(strLine =>
strLine.split(",").map(tryParseFloat)
) persist(StorageLevel.MEMORY_ONLY_SER) // this here!
val grouped: RDD[(Float, Iterable[Array[Float]])] =
input.groupBy(_.last)
On Mon, Oct 13, 2014 at 1:37 PM, Sean Owen <[email protected]> wrote:
> A CSV element like "3.2," takes 4 bytes as text on disk, but, as a
> Double will always take 8 bytes. Is your input like this? that could
> explain it.
>
> You can map to Float in this case to halve the memory, if that works
> for your use case. This is just kind of how Strings and floating-point
> work in the JVM, nothing Spark-specific.
>
> On Mon, Oct 13, 2014 at 9:12 PM, Aris <[email protected]> wrote:
> > Hi guys,
> >
> > I am trying just parse out values from a CSV, everything is a numeric
> > (Double) value, and the input text CSV data is about 1.3 GB in size.
> >
> > When inspect the Java Heap space used by SparkSubmit using JVisualiser
> VM, I
> > end up eating up 8GB of memory! Moreover, by inspecting the BlockManager
> > input, the original 1.3 GB input CSV file goes into an
> RDD[Array[Double]],
> > which is now about 3.6 GB in size from the BlockManager debug output.
> >
> > This means just parsing the CSV of doubles into a very primitive
> container
> > format for Spark almost triples the size of the input... what is going on
> > here? It gets worse because the TOTAL heap space taken up is over 8GB as
> I
> > mentioned.
> >
> > From trying to work through Java Heap output and inspecting it with
> Eclipse
> > Memory Analyze and Jvisualizer VM, it looks like the parsed-out Double
> > values end up taking up twice the amount of memory...meaning about 7.2
> GB of
> > memory, which explains most of the Java Heap space.
> >
> > My code is very simple, does anybody know why I am eating up so much
> memory?
> >
> > val input: RDD[Array[Double]] = sc.textFile(inFile).map(strLine =>
> > strLine.split(",").map(_.toDouble)
> > ) cache()
> > val grouped: RDD[(Double, Iterable[Array[Double]])] =
> > input.groupBy(_.last)
> >
> >
> > Thank you!
> > Aris
>