Re: NRT in Solr
Bill, I think all of the improvements can be made, however they are fairly large structural changes that would require perhaps several patches. The other issue is we'll likely land RT this year (or next) and then the cached values need to be appended to as the documents are added, that and they'll be across several DWPTs (see LUCENE-2324). So one could easily do work for per-segment caching, and then need to go back and do per-segment, append caches. I'm not sure caching is needed at all, especially with the recent speed improvements, except for facets which resemble field caches, and probably should be subsumed there. Jason On Wed, Mar 9, 2011 at 8:27 PM, Bill Bell wrote: > So it looks like can handle adding new documents, and expiring old > documents. Updating a document is not part of the game. > This would work well for message boards or tweet type solutions. > > Solr can do this as well directly. Why wouldn't you just improve the > document and facet caching so that when you append there is not a huge hit > to Solr? Also we could add a expiration to documents as well. > > The big issue for me is that when I update Solr I need to replicate that > change quickly to all slaves. If we changed replication to stream to the > slaves in Near Real Time and not have to create a whole new index version, > warming, etc, that would be awesome. That combined with better caching > smarts and we have a near perfect solution. > > Thanks. > > On 3/9/11 3:29 PM, "Smiley, David W." wrote: > >>Zoie adds NRT to Solr: >>http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin >> >>I haven't tried it yet but looks cool. >> >>~ David Smiley >>Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ >> >>On Mar 9, 2011, at 9:01 AM, Jason Rutherglen wrote: >> >>> Jae, >>> >>> NRT hasn't been implemented NRT as of yet in Solr, I think partially >>> because major features such as replication, caching, and uninverted >>> faceting suddenly are no longer viable, eg, it's another round of >>> testing etc. It's doable, however I think the best approach is a >>> separate request call path, to avoid altering to current [working] >>> API. >>> >>> On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: >>>> Hi, >>>> Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could >>>>not >>>> find the configuration for NRT. >>>> >>>> Regards >>>> >>>> Jae >>>> >> >> >> >> >> > > >
Re: NRT in Solr
Please start new threads for new conversations. On Wed, Mar 9, 2011 at 2:27 AM, stockii wrote: > question: http://wiki.apache.org/solr/NearRealtimeSearchTuning > > > 'PERFORMANCE WARNING: Overlapping onDeckSearchers=x > > i got this message. > in my solrconfig.xml: maxWarmingSearchers=4, if i set this to 1 or 2 i got > exception. with 4 i got nothing, but the Performance Warning. the > wiki-articel says, that the best solution is to set the warmingSearcher to > 1!!! how can this work ? > > - > --- System > > > One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, > 1 Core with 31 Million Documents other Cores < 100.000 > > - Solr1 for Search-Requests - commit every Minute - 5GB Xmx > - Solr2 for Update-Request - delta every Minute - 4GB Xmx > -- > View this message in context: > http://lucene.472066.n3.nabble.com/NRT-in-Solr-tp2652689p2654696.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Lance Norskog goks...@gmail.com
Re: NRT in Solr
So it looks like can handle adding new documents, and expiring old documents. Updating a document is not part of the game. This would work well for message boards or tweet type solutions. Solr can do this as well directly. Why wouldn't you just improve the document and facet caching so that when you append there is not a huge hit to Solr? Also we could add a expiration to documents as well. The big issue for me is that when I update Solr I need to replicate that change quickly to all slaves. If we changed replication to stream to the slaves in Near Real Time and not have to create a whole new index version, warming, etc, that would be awesome. That combined with better caching smarts and we have a near perfect solution. Thanks. On 3/9/11 3:29 PM, "Smiley, David W." wrote: >Zoie adds NRT to Solr: >http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin > >I haven't tried it yet but looks cool. > >~ David Smiley >Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ > >On Mar 9, 2011, at 9:01 AM, Jason Rutherglen wrote: > >> Jae, >> >> NRT hasn't been implemented NRT as of yet in Solr, I think partially >> because major features such as replication, caching, and uninverted >> faceting suddenly are no longer viable, eg, it's another round of >> testing etc. It's doable, however I think the best approach is a >> separate request call path, to avoid altering to current [working] >> API. >> >> On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: >>> Hi, >>> Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could >>>not >>> find the configuration for NRT. >>> >>> Regards >>> >>> Jae >>> > > > > >
Re: NRT in Solr
Jonathan, they have a Wiki up these somewhere, including pretty diagrams. If you have Lucene in Action, Zoie is one of the case studies and is described in a lot of detail. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Jonathan Rochkind > To: "solr-user@lucene.apache.org" > Cc: "Smiley, David W." > Sent: Wed, March 9, 2011 5:34:01 PM > Subject: Re: NRT in Solr > > Interesting, does anyone have a summary of what techniques zoie uses to > do this? I don't see any docs on the technical details. > > On 3/9/2011 5:29 PM, Smiley, David W. wrote: > > Zoie adds NRT to Solr: > > http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin > > > > I haven't tried it yet but looks cool. > > > > ~ David Smiley > > Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ > > > > On Mar 9, 2011, at 9:01 AM, Jason Rutherglen wrote: > > > >> Jae, > >> > >> NRT hasn't been implemented NRT as of yet in Solr, I think partially > >> because major features such as replication, caching, and uninverted > >> faceting suddenly are no longer viable, eg, it's another round of > >> testing etc. It's doable, however I think the best approach is a > >> separate request call path, to avoid altering to current [working] > >> API. > >> > >> On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: > >>> Hi, > >>> Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not > >>> find the configuration for NRT. > >>> > >>> Regards > >>> > >>> Jae > >>> > > > > > > > > > > >
Re: NRT in Solr
Interesting, does anyone have a summary of what techniques zoie uses to do this? I don't see any docs on the technical details. On 3/9/2011 5:29 PM, Smiley, David W. wrote: Zoie adds NRT to Solr: http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin I haven't tried it yet but looks cool. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ On Mar 9, 2011, at 9:01 AM, Jason Rutherglen wrote: Jae, NRT hasn't been implemented NRT as of yet in Solr, I think partially because major features such as replication, caching, and uninverted faceting suddenly are no longer viable, eg, it's another round of testing etc. It's doable, however I think the best approach is a separate request call path, to avoid altering to current [working] API. On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: Hi, Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not find the configuration for NRT. Regards Jae
Re: NRT in Solr
Zoie adds NRT to Solr: http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin I haven't tried it yet but looks cool. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ On Mar 9, 2011, at 9:01 AM, Jason Rutherglen wrote: > Jae, > > NRT hasn't been implemented NRT as of yet in Solr, I think partially > because major features such as replication, caching, and uninverted > faceting suddenly are no longer viable, eg, it's another round of > testing etc. It's doable, however I think the best approach is a > separate request call path, to avoid altering to current [working] > API. > > On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: >> Hi, >> Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not >> find the configuration for NRT. >> >> Regards >> >> Jae >>
Re: NRT in Solr
Jae, NRT hasn't been implemented NRT as of yet in Solr, I think partially because major features such as replication, caching, and uninverted faceting suddenly are no longer viable, eg, it's another round of testing etc. It's doable, however I think the best approach is a separate request call path, to avoid altering to current [working] API. On Tue, Mar 8, 2011 at 1:27 PM, Jae Joo wrote: > Hi, > Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not > find the configuration for NRT. > > Regards > > Jae >
Re: NRT in Solr
maxWarmingSearcher=1 is good for current stable Solr versions where memory is important. Overlapping warming searchers can be extremely memory consuming. I don't know how cache warming behaves with NRT. On Wednesday 09 March 2011 11:27:39 stockii wrote: > question: http://wiki.apache.org/solr/NearRealtimeSearchTuning > > > 'PERFORMANCE WARNING: Overlapping onDeckSearchers=x > > i got this message. > in my solrconfig.xml: maxWarmingSearchers=4, if i set this to 1 or 2 i got > exception. with 4 i got nothing, but the Performance Warning. the > wiki-articel says, that the best solution is to set the warmingSearcher to > 1!!! how can this work ? > > - > --- System > > > One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, > 1 Core with 31 Million Documents other Cores < 100.000 > > - Solr1 for Search-Requests - commit every Minute - 5GB Xmx > - Solr2 for Update-Request - delta every Minute - 4GB Xmx > -- > View this message in context: > http://lucene.472066.n3.nabble.com/NRT-in-Solr-tp2652689p2654696.html Sent > from the Solr - User mailing list archive at Nabble.com. -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
Re: NRT in Solr
question: http://wiki.apache.org/solr/NearRealtimeSearchTuning 'PERFORMANCE WARNING: Overlapping onDeckSearchers=x i got this message. in my solrconfig.xml: maxWarmingSearchers=4, if i set this to 1 or 2 i got exception. with 4 i got nothing, but the Performance Warning. the wiki-articel says, that the best solution is to set the warmingSearcher to 1!!! how can this work ? - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other Cores < 100.000 - Solr1 for Search-Requests - commit every Minute - 5GB Xmx - Solr2 for Update-Request - delta every Minute - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/NRT-in-Solr-tp2652689p2654696.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: NRT in Solr
i am using solr for NRT with this version of solr ... Solr Specification Version: 4.0.0.2010.10.26.08.43.14 Solr Implementation Version: 4.0-2010-10-26_08-05-39 1027394 - hudson - 2010-10-26 08:43:14 Lucene Specification Version: 4.0-2010-10-26_08-05-39 Lucene Implementation Version: 4.0-2010-10-26_08-05-39 1027394 - 2010-10-26 08:43:44 is this version ready for NRT or not ? it works, but if it can work better i gonna be update solr ... thx - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other Cores < 100.000 - Solr1 for Search-Requests - commit every Minute - 4GB Xmx - Solr2 for Update-Request - delta every 2 Minutes - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/NRT-in-Solr-tp2652689p2654472.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: NRT in Solr
I think once this starts yielding matches: trunk/solr$ find . -name \*java | xargs grep IndexReader | grep IndexWriter ...we'll know NRT has landed. Until then: http://wiki.apache.org/solr/NearRealtimeSearch Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Jae Joo > To: solr-user@lucene.apache.org > Sent: Tue, March 8, 2011 4:27:41 PM > Subject: NRT in Solr > > Hi, > Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not > find the configuration for NRT. > > Regards > > Jae >
Re: NRT in Solr
I think once this starts yielding matches: Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Jae Joo > To: solr-user@lucene.apache.org > Sent: Tue, March 8, 2011 4:27:41 PM > Subject: NRT in Solr > > Hi, > Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not > find the configuration for NRT. > > Regards > > Jae >
NRT in Solr
Hi, Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not find the configuration for NRT. Regards Jae
