Hi, We initially went with Hadoop path, but as it is one more software based file system on top of the OS file system, we didn't get a buy in from our system Engineers. i.e In case if we run into any HDFS issues, SEs won't be supporting us :(
Regards, sS --- On Thu, 8/6/09, Walter Underwood <wun...@wunderwood.org> wrote: > From: Walter Underwood <wun...@wunderwood.org> > Subject: Re: Limit of Index size per machine.. > To: solr-user@lucene.apache.org > Date: Thursday, August 6, 2009, 5:12 AM > That is why people don't use search > engines to manage logs. Look at a > Hadoop cluster. > > wunder > > On Aug 5, 2009, at 10:08 PM, Silent Surfer wrote: > > > > > Hi, > > > > That means we need approximately 3000 GB (Index > Size)/24 GB (RAM) = > > 125 servers. > > > > It would be very hard to convince my org to go for 125 > servers for > > log management of 3 Terabytes of indexes. > > > > Has any one used, solr for processing and handling of > the indexes of > > the order of 3 TB ? If so how many servers were used > for indexing > > alone. > > > > Thanks, > > sS > > > > > > --- On Wed, 8/5/09, Ian Connor <ian.con...@gmail.com> > wrote: > > > >> From: Ian Connor <ian.con...@gmail.com> > >> Subject: Re: Limit of Index size per machine.. > >> To: solr-user@lucene.apache.org > >> Date: Wednesday, August 5, 2009, 9:38 PM > >> I try to keep the index directory > >> size less than the amount of RAM and rely > >> on the OS to cache as it needs. Linux does a > pretty good > >> job here and I am > >> sure OS X will do a good job also. > >> > >> Distributed search here will be your friend so you > can > >> chunk it up to a > >> number of servers to keep your cost down (2GB RAM > sticks > >> are much cheaper > >> than 4GB RAM sticks $20 < $100). > >> > >> Ian. > >> > >> On Wed, Aug 5, 2009 at 1:44 PM, Silent Surfer > <silentsurfe...@yahoo.com > > >> >wrote: > >> > >>> > >>> Hi , > >>> > >>> We are planning to use Solr for indexing the > server > >> log contents. > >>> The expected processed log file size per day: > 100 GB > >>> We are expecting to retain these indexes for > 30 days > >> (100*30 ~ 3 TB). > >>> > >>> Can any one provide what would be the optimal > size of > >> the index that I can > >>> store on a single server, without hampering > the search > >> performance etc. > >>> > >>> We are planning to use OSX server with a > configuration > >> of 16 GB (Can go to > >>> 24 GB). > >>> > >>> We need to figure out how many servers are > required to > >> handle such amount > >>> of data.. > >>> > >>> Any help would be greatly appreciated. > >>> > >>> Thanks > >>> SilentSurfer > >>> > >>> > >>> > >>> > >>> > >> > >> > >> -- > >> Regards, > >> > >> Ian Connor > >> 1 Leighton St #723 > >> Cambridge, MA 02141 > >> Call Center Phone: +1 (714) 239 3875 (24 hrs) > >> Fax: +1(770) 818 5697 > >> Skype: ian.connor > >> > > > > > > > > > >
