Be interested to know what that is, if you don't mind sharing.
We're thinking of deploying a Ceph cluster for another project anyway,
it seems to remove some of the chokepoints/points of failure HDFS suffers from
but I've no idea how well it can interoperate with the usual HDFS clients
(Spark in
We use lustre and a couple internal data storage services. I wouldn't
recommend lustre much; it's got an SPOF which is a problem at scale. I just
wanted to point out that you can skip hdfs if you so choose.
On Wednesday, October 22, 2014, Dick Davies d...@hellooperator.net wrote:
Be interested
, October 22, 2014 2:29:20 AM
Subject: Re: Do i really need HDFS?
Be interested to know what that is, if you don't mind sharing.
We're thinking of deploying a Ceph cluster for another project anyway,
it seems to remove some of the chokepoints/points of failure HDFS suffers
from
but I've no idea
Ok so,
I'd be curious to know your final architecture (D. Davies)?
I was looking to put Ceph on top of the (3) btrfs nodes in case we need
a DFS at some later point. We're not really sure what softwares will be
in our final mix. Certainly installing Ceph does not hurt anything (?);
and I'm
We use spark without HDFS--in our case, we just use ansible to copy the
spark executors onto all hosts at the same path. We also load and store our
spark data from non-HDFS sources.
On Tue, Oct 21, 2014 at 4:57 AM, Dick Davies d...@hellooperator.net wrote:
I think Spark needs a way to send jobs
Message -
From: Ankur Chauhan an...@malloc64.com
To: user@mesos.apache.org
Sent: Tuesday, October 21, 2014 10:28:50 AM
Subject: Re: Do i really need HDFS?
This is what I also intend to do. Is a s3 path considered non-hdfs? If so,
how does it know the credentials to use to fetch the file
Hi all,
I am trying to setup a new mesos cluster and I so far I have a set of master
and slave nodes working and I can get everything running. I am able to install
and run a couple of sample apps, hookup jenkins etc. My main question now is Do
I really need HDFS? All my artifacts (for apps
far I have a set of
master and slave nodes working and I can get everything running. I am able
to install and run a couple of sample apps, hookup jenkins etc. My main
question now is Do I really need HDFS? All my artifacts (for apps) are on a
protected S3 bucket or in a private docker registry
all,
I am trying to setup a new mesos cluster and I so far I have a set
of master and slave nodes working and I can get everything running.
I am able to install and run a couple of sample apps, hookup jenkins
etc. My main question now is Do I really need HDFS? All my artifacts
9 matches
Mail list logo
