I think Spark needs a way to send jobs to/from the workers - the Spark distro itself will pull down the executor ok, but in my (very basic) tests I got stuck without HDFS.
So basically it depends on the framework. I think in Sparks case they assume most users are migrating from an existing Hadoop deployment, so HDFS is sort of assumed. On 20 October 2014 23:18, CCAAT <[email protected]> wrote: > On 10/20/14 11:46, Steven Schlansker wrote: > > >> We are running Mesos entirely without HDFS with no problems. We use >> Docker to distribute our >> application to slave nodes, and keep no state on individual nodes. > > > > Background: I'm building up a 3 node cluster to run mesos and spark. No > legacy Hadoop needed or wanted. I am using btrfs for the local file system, > with (2) drives set up for raid1 on each system. > > So you are suggesting that I can install mesos + spark + docker > and not a DFS on these (3) machines? > > > Will I need any other softwares? My application is a geophysical > fluid simulator, so scala, R, and all sorts of advanced math will > be required on the cluster for the Finite Element Methods. > > > James > >
