Hi Dave, Thanks for your feedback.
On Fri, Dec 17, 2010 at 11:14 PM, Buttler, David <[email protected]> wrote: > For a small cluster like you have, I would be fine with putting all of the > master pieces on one node: > ZK, NN, HMaster, ganglia, nagios. > Thanks for the confirmation. > You say you are going to put solr on other nodes -- that makes sense as solr > can be very memory hungry, and you might not want to share 12 GB of RAM > between solr and hbase. On the other hand, I have no problem with > collocating solr and hbase on a large memory machine -- as long as there is > no chance of swapping. The only problems come up when you run a heavy > map/reduce job over the nodes and kill disk and network i/o -- that can slow > solr down. But, as JD often recommends, it is better to keep your developers > off of your production system and run your map/reduce jobs on a dedicated > cluster (if you can afford the space/power/cooling/$ for multiple clusters). > Currently, we do not have any M/R Job to go with, but the plan is you mentioned and JD recommends production servers will be off-limit for M/R and for that we have a development cluster. Regards, Imran > Dave > > -----Original Message----- > From: Imran M Yousuf [mailto:[email protected]] > Sent: Friday, December 17, 2010 4:40 AM > To: [email protected] > Subject: Re: [RFC] Deployment layout and server configurations > > Hi JG :), > > On Fri, Dec 17, 2010 at 12:11 PM, Jonathan Gray <[email protected]> wrote: >> Hey Imran, >> >> This looks reasonable but it's hard to say without knowing what the >> read/write workload is like. You say all searches are done using Solr... >> will that also be hosted on these servers? >> > > No Solr will not be collocated with them :), I am glad you remember me > mentioning about it initially, I have attached the full System diagram > in this email. > >> One thing. It looks like you have two servers for ZK? ZK should always be >> run in odd numbers (1, 3, 5, or 7). For a cluster of this size, you are >> already sensitive to a failure of the NN so you could just put ZK on the >> master node. It has RAID so you won't be sensitive to a single disk >> failure. Otherwise 3 nodes. > > Thanks for suggestion, how about if I put Gangrila and Nagios with the > NN and MN as well? > > Regards, > >> >> JG >> >>> -----Original Message----- >>> From: Imran M Yousuf [mailto:[email protected]] >>> Sent: Thursday, December 16, 2010 9:04 PM >>> To: [email protected] >>> Subject: [RFC] Deployment layout and server configurations >>> >>> Hi, >>> >>> I will go straight to the point, we are purchasing hardwares for going live. >>> Currently we are targeting a data capacity of ~5TB with a view to increase >>> capacity as the load increases. I have attached a tentative schematic as >>> attachment. Please give us feedback on the network organization and >>> additional tools as you see necessary. >>> >>> * The HBase cluster will be in a separate 1Gbps network, connected through >>> a managed switch. >>> * Currently we do not have any MapReduce jobs, only Gets and occasional >>> Scans. All searches are done using Solr. Application layer caching will >>> also be >>> in place. >>> * 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X >>> X3440/E5560 (4C/8T) >>> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X >>> X3440/E5560 (4C/8T) >>> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T) >>> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560 >>> (4C/8T) >>> >>> It would really be helpful for me if you would kindly help us on whether >>> these configurations are sufficient or not. The nature of the application >>> running is a E-Commerce PaaS initially. Later when we add additional >>> applications to the stack we will be able to increase the capacity. >>> >>> Regards, >>> >>> -- >>> Imran M Yousuf >>> Twitter: @imyousuf - http://twitter.com/imyousuf >>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/ >>> Mobile: +880-1711402557 >> > > > > -- > Imran M Yousuf > Entrepreneur & CEO > Smart IT Engineering Ltd. > Dhaka, Bangladesh > Twitter: @imyousuf - http://twitter.com/imyousuf > Blog: http://imyousuf-tech.blogs.smartitengineering.com/ > Mobile: +880-1711402557 > -- Imran M Yousuf Entrepreneur & CEO Smart IT Engineering Ltd. Dhaka, Bangladesh Twitter: @imyousuf - http://twitter.com/imyousuf Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557
