Yes, actually even if you don’t set it to true, on-disk data is compressed. (This setting only affects serialized data in memory).
Matei On Jun 11, 2014, at 2:56 PM, Surendranauth Hiraman <suren.hira...@velos.io> wrote: > Hi, > > Will spark.rdd.compress=true enable compression when using DISK_ONLY > persistence? > > > > SUREN HIRAMAN, VP TECHNOLOGY > Velos > Accelerating Machine Learning > > 440 NINTH AVENUE, 11TH FLOOR > NEW YORK, NY 10001 > O: (917) 525-2466 ext. 105 > F: 646.349.4063 > E: suren.hira...@velos.io > W: www.velos.io >
