If it's locally mounted via fuse then there is no issue. Also there are tickets open about volume mounting in the sandbox, that would be the ideal solution.
Cheers, Tim ----- Original Message ----- > From: "Dick Davies" <[email protected]> > To: [email protected] > Sent: Wednesday, October 22, 2014 2:29:20 AM > Subject: Re: Do i really need HDFS? > > Be interested to know what that is, if you don't mind sharing. > > We're thinking of deploying a Ceph cluster for another project anyway, > it seems to remove some of the chokepoints/points of failure HDFS suffers > from > but I've no idea how well it can interoperate with the usual HDFS clients > (Spark in my particular case but I'm trying to keep this general). > > On 21 October 2014 13:16, David Greenberg <[email protected]> wrote: > > We use spark without HDFS--in our case, we just use ansible to copy the > > spark executors onto all hosts at the same path. We also load and store our > > spark data from non-HDFS sources. > > > > On Tue, Oct 21, 2014 at 4:57 AM, Dick Davies <[email protected]> > > wrote: > >> > >> I think Spark needs a way to send jobs to/from the workers - the Spark > >> distro itself > >> will pull down the executor ok, but in my (very basic) tests I got > >> stuck without HDFS. > >> > >> So basically it depends on the framework. I think in Sparks case they > >> assume most > >> users are migrating from an existing Hadoop deployment, so HDFS is > >> sort of assumed. > >> > >> > >> On 20 October 2014 23:18, CCAAT <[email protected]> wrote: > >> > On 10/20/14 11:46, Steven Schlansker wrote: > >> > > >> > > >> >> We are running Mesos entirely without HDFS with no problems. We use > >> >> Docker to distribute our > >> >> application to slave nodes, and keep no state on individual nodes. > >> > > >> > > >> > > >> > Background: I'm building up a 3 node cluster to run mesos and spark. No > >> > legacy Hadoop needed or wanted. I am using btrfs for the local file > >> > system, > >> > with (2) drives set up for raid1 on each system. > >> > > >> > So you are suggesting that I can install mesos + spark + docker > >> > and not a DFS on these (3) machines? > >> > > >> > > >> > Will I need any other softwares? My application is a geophysical > >> > fluid simulator, so scala, R, and all sorts of advanced math will > >> > be required on the cluster for the Finite Element Methods. > >> > > >> > > >> > James > >> > > >> > > > > > > -- Cheers, Timothy St. Clair Red Hat Inc.
