If HDFS is not required what is the minimum config needed to fetch s3:// urns?
Sent from my iPhone > On Oct 20, 2014, at 7:53 AM, David Greenberg <[email protected]> wrote: > > A DFS is merely a convenience for getting data to all the nodes of your mesos > cluster. If you want to have all nodes retrieve data from HTTP servers (like > S3), this is feasible as well. If you want your nodes to store data, you can > have them write to any database (although I'd recommend something scalable, > like Riak, rather than a DB like an unsharded MySQL) > > On Mon, Oct 20, 2014 at 11:33 AM, CCAAT <[email protected]> wrote: >> If one is building a mesos-cluster, then is a DFS mandatory, or is there >> combinations of other codes that suffice for the needs of a mesos cluster? >> >> So what is the list of available Distributed File Systems that are generally >> available and mesos is know to work on top of? >> >> Glusterfs, Lustrefs, FhGFS (BeeGFS), Ceph >> >> [1] http://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems >> >> >> An explicit list of other DFS's or combinations of codes that provide >> these required 'feature sets' is of interest to many. >> >> >> James >> >> >> >> >>> On 10/20/14 07:08, David Greenberg wrote: >>> You certainly don't need hdfs if you've got other infrastructure that's >>> providing the same sorts of features. We don't run hdfs on our mesos >>> cluster, and it's fine. >>> >>> On Monday, October 20, 2014, Ankur Chauhan <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> __ >>> Hi all, >>> >>> I am trying to setup a new mesos cluster and I so far I have a set >>> of master and slave nodes working and I can get everything running. >>> I am able to install and run a couple of sample apps, hookup jenkins >>> etc. My main question now is Do I really need HDFS? All my artifacts >>> (for apps) are on a protected S3 bucket or in a private docker registry. >>> >>> If I need HDFS, do I need to go "all in" even when I am not using >>> hdfs as a data store but rather as a simple way to fetch files from >>> s3; or can I get away with putting the correct core-site.xml and >>> hdfs-site.xml in HADOOP_HOME and get away with it? >>> >>> It would really help how other have their mesos setup in production >>> or what they would recommend regarding my setup? >>> >>> >>> -- >>> Ankur Chauhan >
