Hi,
In one of the application we have made which had no clone stuff, we have
set the value of spark.storage.memoryFraction to very low, and yes that
gave us performance benefits.
Regarding that issue, you should also look at the data you are trying to
broadcast, as sometimes creating that data structure at executor's itself
as singleton helps.
Thanks,
On Tue, Apr 14, 2015 at 12:23 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
You could try leaving all the configuration values to default and running
your application and see if you are still hitting the heap issue, If so try
adding a Swap space to the machines which will definitely help. Another way
would be to set the heap space manually (export _JAVA_OPTIONS=-Xmx5g)
Thanks
Best Regards
On Wed, Apr 8, 2015 at 12:45 AM, Shuai Zheng szheng.c...@gmail.com
wrote:
Hi All,
I am a bit confused on spark.storage.memoryFraction, this is used to set
the area for RDD usage, will this RDD means only for cached and persisted
RDD? So if my program has no cached RDD at all (means that I have no
.cache() or .persist() call on any RDD), then I can set this
spark.storage.memoryFraction to a very small number or even zero?
I am writing a program which consume a lot of memory (broadcast value,
runtime, etc). But I have no cached RDD, so should I just turn off this
spark.storage.memoryFraction to 0 (which will help me to improve the
performance)?
And I have another issue on the broadcast, when I try to get a broadcast
value, it throws me out of memory error, which part of memory should I
allocate more (if I can’t increase my overall memory size).
java.lang.OutOfMemoryError: Java heap spac
e
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA
rraySerializer.read(DefaultArraySerializers.java:218)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA
rraySerializer.read(DefaultArraySerializers.java:200)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699)
at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea
d(FieldSerializer.java:611)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria
lizer.java:221)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea
d(FieldSerializer.java:605)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria
lizer.java:221)
at
com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at
org.apache.spark.serializer.KryoDeserializationStream.readObject(Kryo
Serializer.scala:138)
at
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Ser
ializer.scala:133)
at
org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
at
org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:2
48)
at
org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:13
6)
at
org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:5
49)
at
org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:431
)
at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlo
ck$1.apply(TorrentBroadcast.scala:167)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152)
at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(Torren
tBroadcast.scala:164)
at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(Torrent
Broadcast.scala:64)
at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.s
cala:64)
at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast
.scala:87)
Regards,
Shuai
