It doesn't mean 'out of memory'; it means 'you can't allocate a byte[]
over 2GB in the JVM'. Something is serializing a huge block somewhere.
there are a number of related JIRAs and discussions on JIRA and this
mailing list; have a browse of those first for back story.

On Mon, Feb 23, 2015 at 6:44 PM, insperatum <[email protected]> wrote:
> Hi, I'm using MLLib to train a random forest. It's working fine to depth 15,
> but if I use depth 20 I get a java.lang.OutOfMemoryError: Requested array
> size exceeds VM limit on the driver, from the collectAsMap operation in
> DecisionTree.scala, around line 642. It doesn't happen until a good hour
> into training. I'm using 50 treees on 36 slaves with maxMemoryInMB=250, but
> still get an error even if I use a driver memory of 240G. Has anybody seen
> this error in this context before, and can advise on what might be
> triggering it? Best, Luke
> ________________________________
> View this message in context: Requested array size exceeds VM limit
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to