Hi Craig, The best configuration is to have multiple disks configured as separate filesystems (so no RAID), and set the spark.local.dir property, which configures Spark's scratch space directories, to be a comma-separated list of directories, one per disk. In 0.8 we've written a bit on how to configure machines for Spark here: http://spark.incubator.apache.org/docs/latest/hardware-provisioning.html. For the filesystem I'd suggest ext3 with noatime set.
Matei On Oct 14, 2013, at 11:28 AM, Craig Vanderborgh <[email protected]> wrote: > Hi All, > > We're setting up a new Spark-on-Mesos cluster. I'd like anyone who is > already done this to suggest a disk partitioning/filesystem layout that has > worked well for them in their cluster deployment. > > We are running MapR M3 on the cluster, but only for maprfs. Our jobs will be > programmed for and run on Spark. > > Thanks in advance, > Craig Vanderborgh
