Re: Separating Index Reader and Writer
Push again. Regards Em wrote: Just wanted to push that topic. Regards Em wrote: Hi Peter, I must jump in this discussion: From a logical point of view what you are saying makes only sense if both instances do not run on the same machine or at least not on the same drive. When both run on the same machine and the same drive, the overall used memory should be equal plus I do not understand why this setup should affect cache warming etc., since the process of rewarming should be the same. Well, my knowledge about the internals is not very deep. But from just a logical point of view - to me - the same is happening as if I would do it in a single solr-instance. So what is the difference, what do I overlook? Another thing: While W is committing and writing to the index, is there any inconsistency in R or isn't there any, because W is writing a new Segment and so for R there isn't anything different until the commit finished? Are there problems during optimizing an index? How do you inform R about the finished commit? Thank you for your explanation, it's a really interesting topic! Regards, Em Peter Sturge-2 wrote: Hi, We use this scenario in production where we have one write-only Solr instance and 1 read-only, pointing to the same data. We do this so we can optimize caching/etc. for each instance for write/read. The main performance gain is in cache warming and associated parameters. For your Index W, it's worth turning off cache warming altogether, so commits aren't slowed down by warming. Peter On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia isan.fu...@germinait.com wrote: Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia. -- View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2516736.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Separating Index Reader and Writer
Just wanted to push that topic. Regards Em wrote: Hi Peter, I must jump in this discussion: From a logical point of view what you are saying makes only sense if both instances do not run on the same machine or at least not on the same drive. When both run on the same machine and the same drive, the overall used memory should be equal plus I do not understand why this setup should affect cache warming etc., since the process of rewarming should be the same. Well, my knowledge about the internals is not very deep. But from just a logical point of view - to me - the same is happening as if I would do it in a single solr-instance. So what is the difference, what do I overlook? Another thing: While W is committing and writing to the index, is there any inconsistency in R or isn't there any, because W is writing a new Segment and so for R there isn't anything different until the commit finished? Are there problems during optimizing an index? How do you inform R about the finished commit? Thank you for your explanation, it's a really interesting topic! Regards, Em Peter Sturge-2 wrote: Hi, We use this scenario in production where we have one write-only Solr instance and 1 read-only, pointing to the same data. We do this so we can optimize caching/etc. for each instance for write/read. The main performance gain is in cache warming and associated parameters. For your Index W, it's worth turning off cache warming altogether, so commits aren't slowed down by warming. Peter On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia isan.fu...@germinait.com wrote: Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia. -- View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2452238.html Sent from the Solr - User mailing list archive at Nabble.com.
Separating Index Reader and Writer
Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia.
Re: Separating Index Reader and Writer
Hi, We use this scenario in production where we have one write-only Solr instance and 1 read-only, pointing to the same data. We do this so we can optimize caching/etc. for each instance for write/read. The main performance gain is in cache warming and associated parameters. For your Index W, it's worth turning off cache warming altogether, so commits aren't slowed down by warming. Peter On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia isan.fu...@germinait.com wrote: Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia.
Re: Separating Index Reader and Writer
Hi peter , Can you elaborate a little on how performance gain is in cache warming.I am getting a good improvement on search time. On 6 February 2011 23:29, Peter Sturge peter.stu...@gmail.com wrote: Hi, We use this scenario in production where we have one write-only Solr instance and 1 read-only, pointing to the same data. We do this so we can optimize caching/etc. for each instance for write/read. The main performance gain is in cache warming and associated parameters. For your Index W, it's worth turning off cache warming altogether, so commits aren't slowed down by warming. Peter On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia isan.fu...@germinait.com wrote: Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia. -- Thanks Regards, Isan Fulia.
Re: Separating Index Reader and Writer
Hi Peter, I must jump in this discussion: From a logical point of view what you are saying makes only sense if both instances do not run on the same machine or at least not on the same drive. When both run on the same machine and the same drive, the overall used memory should be equal plus I do not understand why this setup should affect cache warming etc., since the process of rewarming should be the same. Well, my knowledge about the internals is not very deep. But from just a logical point of view - to me - the same is happening as if I would do it in a single solr-instance. So what is the difference, what do I overlook? Another thing: While W is committing and writing to the index, is there any inconsistency in R or isn't there any, because W is writing a new Segment and so for R there isn't anything different until the commit finished? Are there problems during optimizing an index? How do you inform R about the finished commit? Thank you for your explanation, it's a really interesting topic! Regards, Em Peter Sturge-2 wrote: Hi, We use this scenario in production where we have one write-only Solr instance and 1 read-only, pointing to the same data. We do this so we can optimize caching/etc. for each instance for write/read. The main performance gain is in cache warming and associated parameters. For your Index W, it's worth turning off cache warming altogether, so commits aren't slowed down by warming. Peter On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia isan.fu...@germinait.com wrote: Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via dataDir). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks Regards, Isan Fulia. -- View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2438730.html Sent from the Solr - User mailing list archive at Nabble.com.
