Hi Matei, This is helpful but it would be even more so if this documentation could describe how to make these settings correctly in a Spark-on-Mesos environment. Can you describe the differences for Mesos?
Thanks again, Craig On Mon, Oct 14, 2013 at 6:15 PM, Matei Zaharia <[email protected]>wrote: > Hi Craig, > > The best configuration is to have multiple disks configured as separate > filesystems (so no RAID), and set the spark.local.dir property, which > configures Spark's scratch space directories, to be a comma-separated list > of directories, one per disk. In 0.8 we've written a bit on how to > configure machines for Spark here: > http://spark.incubator.apache.org/docs/latest/hardware-provisioning.html. > For the filesystem I'd suggest ext3 with noatime set. > > Matei > > On Oct 14, 2013, at 11:28 AM, Craig Vanderborgh < > [email protected]> wrote: > > > Hi All, > > > > We're setting up a new Spark-on-Mesos cluster. I'd like anyone who is > already done this to suggest a disk partitioning/filesystem layout that has > worked well for them in their cluster deployment. > > > > We are running MapR M3 on the cluster, but only for maprfs. Our jobs > will be programmed for and run on Spark. > > > > Thanks in advance, > > Craig Vanderborgh > >
