Hello ; I checked on my partitions/storage and here is what I have I have 80 executors 5 G per executore.
Do i need to set additional params say cores spark.serializer org.apache.spark.serializer.KryoSerializer # spark.driver.memory 5g # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" spark.shuffle.memoryFraction 0.3 spark.storage.memoryFraction 0.65 RDD Name Storage Level Cached Partitions Fraction Cached Size in Memory Size in Tachyon Size on Disk ratingBlocks Memory Deserialized 1x Replicated 257 129% 4.1 GB 0.0 B 0.0 B itemOutBlocks Memory Deserialized 1x Replicated 100 100% 7.3 MB 0.0 B 0.0 B 38 Memory Serialized 1x Replicated 193 97% 5.6 GB 0.0 B 0.0 B userInBlocks Memory Deserialized 1x Replicated 100 100% 2.8 GB 0.0 B 0.0 B itemFactors-1 Memory Deserialized 1x Replicated 69 69% 8.4 MB 0.0 B 0.0 B itemInBlocks Memory Deserialized 1x Replicated 69 69% 1455.3 MB 0.0 B 0.0 B userFactors-1 Memory Deserialized 1x Replicated 100 100% 35.0 GB 0.0 B 0.0 B userOutBlocks Memory Deserialized 1x Replicated 100 100% 1062.7 MB 0.0 B 0.0 B On Jun 26, 2015, at 8:26 AM, Xiangrui Meng <men...@gmail.com> wrote: > number of CPU cores or less.
