Thanks Furkam, This give some really good understanding. We have Amazon Instance and right now it is running on m1.large.
In Amazon we are not finding a support to increase ONLY RAM ! that our main concern and we are actively looking which instance can help us to support this index size. Do you think m2.4xlarge would help?. Also if you can share some input over speeding up the indexer process?. Regards, Priti On Mon, Mar 10, 2014 at 6:30 PM, Furkan KAMACI <furkankam...@gmail.com>wrote: > Hi Priti; > > 100 qps is not much but 7 GB is too low and it may be a problem for you. I > have a tens of nodes of SolrCloud and I send data them via Map/Reduce via > tens of servers. However indexing speed did not be a problem for me yet. > Problems occurs because of network communication, RAM or something else. > > If we talk about the ideal size of RAM: then you should have a RAM total of > at least your index size. Because you may want to cache everything ideally. > However if you consider the situations as like optimize you it would be > nice 3 times larger of your index size. However RAM is also depends on your > user characteristics. Sometimes caching only some of documents may be > enough because of they are queried many many times. > > As I mentioned previously 100 qps is not much and indexing is not a > bottleneck at many situations I suggest to remember the occam's razor. > Start with simple and increase your RAM step by step. On the other hand I > suggest you to use a tool as like Solrmeter to test your qps. > > If you have any question I can help you about your infrastructure. > > Thanks; > Furkan KAMACI > > > 2014-03-10 12:41 GMT+02:00 Priti Solanki <pritiatw...@gmail.com>: > > > As of now index is on 136 GB. > > > > I want to understand can we do multiple write on solr? I don't have any > > partitioning strategy as of now. > > > > On Amazon instance for Solr the disk read/write a like 5% or so . I am > not > > able to understand even though I am almost processing 300 records per min > > how come Solr server is not receiving high disk IO. > > > > I have a query, When solr is writing does it update complete index again > ? > > > > I will try and increase the RAM to 128 and see if some improvement > comes.. > > > > > > On Mon, Mar 10, 2014 at 9:21 AM, Susheel Kumar < > > susheel.ku...@thedigitalgroup.net> wrote: > > > > > Not sure how fast your index will grow but first you may still want to > > > consider upgrading the single machine to 128 GB to see how the > > performance > > > is coming. Current memory 7 GB is really low. After that you may want > to > > > add another node to partition the index into 2 nodes/shards (assuming > you > > > have some partition strategy ) that index size of 150-200 GB can fit > into > > > two nodes memory. > > > > > > Thanks, > > > Susheel > > > > > > -----Original Message----- > > > From: Priti Solanki [mailto:pritiatw...@gmail.com] > > > Sent: Friday, March 07, 2014 12:50 AM > > > To: solr-user@lucene.apache.org > > > Subject: Re: SolrCloud setup guidance > > > > > > Thanks Susheel, > > > > > > But this index will keep on growing that my worry So I always have to > > > increase the RAM . > > > > > > Can you suggest how many nodes one can think to support this bug index? > > > > > > Regards, > > > > > > > > > > > > On Fri, Mar 7, 2014 at 2:50 AM, Susheel Kumar < > > > susheel.ku...@thedigitalgroup.net> wrote: > > > > > > > Setting up Solr cloud(horizontal scaling) is definitely a good idea > > > > for this big index but before going to Solr cloud, are you able to > > > > upgrade your single node to 128GB of memory(vertical scaling) to see > > the > > > difference. > > > > > > > > Thanks, > > > > Susheel > > > > > > > > -----Original Message----- > > > > From: Priti Solanki [mailto:pritiatw...@gmail.com] > > > > Sent: Thursday, March 06, 2014 10:51 AM > > > > To: solr-user@lucene.apache.org > > > > Subject: SolrCloud setup guidance > > > > > > > > Hello Everyone, > > > > > > > > I would like to take you guidance of following > > > > > > > > I have a single core with 124 GB of index data size. Indexing and > > > > Reading both are very slow as I have 7 GB RAM to support this huge > > > > data. Almost 8 million of documents. > > > > > > > > Hence, we thought of going to SolrCloud so that we can accommodate > > > > more upcoming data. I have data for 13 country with their millions of > > > > products and we want to set up solrcloud for the same. > > > > > > > > I am in need of some initial thoughts about how to setup solrCloud > for > > > > such requirement. How to we come to know how many nodes,core I would > > > > be needing to support this... > > > > > > > > we are thinking to host this on Amazon...Any guidance or reading > links > > > > ,case study will be highly appreciated. > > > > > > > > Regards, > > > > > > > > > >
