Hi PA,
In my environment, we had a SAN storage and I/O was pretty good. So if you
have similar environment then I don't see any performance issues.
Just out of curiosity - what amount of data are you looking forward to process ?
Regards,
Sagar
-----Original Message-----
From: Pierre Antoine Du Bois De Naurois [mailto:pad...@gmail.com]
Sent: Thursday, May 17, 2012 8:29 PM
To: common-user@hadoop.apache.org
Subject: Re: is hadoop suitable for us?
Thanks Sagar, Mathias and Michael for your replies.
It seems we will have to go with hadoop even if I/O will be slow due to our
configuration.
I will try to update on how it worked for our case.
Best,
PA
2012/5/17 Michael Segel <michael_se...@hotmail.com>
> The short answer is yes.
> The longer answer is that you will have to account for the latencies.
>
> There is more but you get the idea..
>
> Sent from my iPhone
>
> On May 17, 2012, at 5:33 PM, "Pierre Antoine Du Bois De Naurois" <
> pad...@gmail.com> wrote:
>
> > We have large amount of text files that we want to process and index
> (plus
> > applying other algorithms).
> >
> > The problem is that our configuration is share-everything while
> > hadoop
> has
> > a share-nothing configuration.
> >
> > We have 50 VMs and not actual servers, and these share a huge
> > central storage. So using HDFS might not be really useful as
> > replication will not help, distribution of files have no meaning as
> > all files will be again located in the same HDD. I am afraid that
> > I/O will be very slow with or without HDFS. So i am wondering if it
> > will really help us to use hadoop/hbase/pig etc. to distribute and
> > do several parallel tasks.. or is "better" to install something
> > different (which i am not sure what). We heard myHadoop is better
> > for such kind of configurations, have any clue about it?
> >
> > For example we now have a central mySQL to check if we have already
> > processed a document and keeping there several metadata. Soon we
> > will
> have
> > to distribute it as there is not enough space in one VM, But
> > Hadoop/HBase will be useful? we don't want to do any complex
> > join/sort of the data, we just want to do queries to check if
> > already processed a document, and if not to add it with several of it's
> > metadata.
> >
> > We heard sungrid for example is another way to go but it's
> > commercial. We are somewhat lost.. so any help/ideas/suggestions are
> > appreciated.
> >
> > Best,
> > PA
> >
> >
> >
> > 2012/5/17 Abhishek Pratap Singh <manu.i...@gmail.com>
> >
> >> Hi,
> >>
> >> For your question if HADOOP can be used without HDFS, the answer is Yes.
> >> Hadoop can be used with any kind of distributed file system.
> >> But I m not able to understand the problem statement clearly to
> >> advice
> my
> >> point of view.
> >> Are you processing text file and saving in distributed database??
> >>
> >> Regards,
> >> Abhishek
> >>
> >> On Thu, May 17, 2012 at 1:46 PM, Pierre Antoine Du Bois De Naurois
> >> < pad...@gmail.com> wrote:
> >>
> >>> We want to distribute processing of text files.. processing of
> >>> large machine learning tasks, have a distributed database as we
> >>> have big
> amount
> >>> of data etc.
> >>>
> >>> The problem is that each VM can have up to 2TB of data (limitation
> >>> of
> >> VM),
> >>> and we have 20TB of data. So we have to distribute the processing,
> >>> the database etc. But all those data will be in a shared huge
> >>> central file system.
> >>>
> >>> We heard about myHadoop, but we are not sure why is that any
> >>> different
> >> from
> >>> Hadoop.
> >>>
> >>> If we run hadoop/mapreduce without using HDFS? is that an option?
> >>>
> >>> best,
> >>> PA
> >>>
> >>>
> >>> 2012/5/17 Mathias Herberts <mathias.herbe...@gmail.com>
> >>>
> >>>> Hadoop does not perform well with shared storage and vms.
> >>>>
> >>>> The question should be asked first regarding what you're trying
> >>>> to
> >>> achieve,
> >>>> not about your infra.
> >>>> On May 17, 2012 10:39 PM, "Pierre Antoine Du Bois De Naurois" <
> >>>> pad...@gmail.com> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> We have about 50 VMs and we want to distribute processing across
> >> them.
> >>>>> However these VMs share a huge data storage system and thus
> >>>>> their
> >>>> "virtual"
> >>>>> HDD are all located in the same computer. Would Hadoop be useful
> >>>>> for
> >>> such
> >>>>> configuration? Could we use hadoop without HDFS? so that we can
> >>> retrieve
> >>>>> and store everything in the same storage?
> >>>>>
> >>>>> Thanks,
> >>>>> PA
> >>>>>
> >>>>
> >>>
> >>
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended
recipient, you are not authorized to read, retain, copy, print, distribute or
use this message. If you have received this communication in error, please
notify the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
