You can also look at the shuffle file cleanup tricks we do inside of the ALS algorithm in Spark.
On Fri, Feb 23, 2018 at 6:20 PM, vijay.bvp <bvpsa...@gmail.com> wrote: > have you looked at > http://apache-spark-user-list.1001560.n3.nabble.com/Limit- > Spark-Shuffle-Disk-Usage-td23279.html > > and the post mentioned there > https://forums.databricks.com/questions/277/how-do-i-avoid- > the-no-space-left-on-device-error.html > > also try compressing the output > https://spark.apache.org/docs/latest/configuration.html# > compression-and-serialization > spark.shuffle.compress > > thanks > Vijay > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Twitter: https://twitter.com/holdenkarau
