Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
: We will take this approach in our production environment but meanwhile I am : curious if this issue will be addressed: it seems the new/first searchers do : not really buy any performance benefits because it uses so much memory, : especially at core loading time. There's nothing inheriently wrong with using newSearcher/firstSearcher -- for many people they do in fact provide a perf improvement for real users (at the cost of some initial time spent warming before those users ever get access to the searcher) As i udnerstand it from this thread, your issue is not actually the firstSearcher/newSearcher -- your issue (per yonik's ocmments) is that with per Segment sorting in 1.4, the FieldCache for some of your fields requires a lot more ram in 1.4 then it would have been for Solr 1.3 -- which caused GC thrashing during initialization. Even w/o using firstSearcher/newSearcher, all that RAM is still going to be used if/when you do sort on those fields -- all removing the firstSearcher/newSearcher queries on those fields has done for you is delay when the time spent initializing those FieldCaches happens and when that RAM first starts getting used. It's posisbly you never actual sort on those fields, in which case removing those warming queries completely is definitely the way to go -- but if you do sort on them then the warming queries can still legitimately be helpful (in thta they pay the cost up front before a real user issues queries) As yonik mentioned the real fix for the amount of memory being used is to switch to the TrieDateFields which use much more efficient FieldCache's for sorting -- with that change you can probably start using the warming queries again. (Depending on how you tested, you may not have noticed much advantage to having them because you'll really only see the advantages on the initial queries that do sorting -- those should show huge outlier times w/o the warming queries, but once those poor unlucky users have paid the price for initializing hte FieldCache, every one elses sorts should be fast) -Hoss
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
just a blind shot (didn't read the full thread): what is your maxWarmingSearchers settings? For large indices we set it to 2 (maximum) Regards, Peter. just update on this issue... we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran benchmark tests, there is no noticeable performance impact on the queries we perform comparing with Solr 1.3 benchmark tests WITH new/first searchers. Also the memory usage reduced by 5.5 GB after loading the cores with our data volume by turning these static warm caches off. We will take this approach in our production environment but meanwhile I am curious if this issue will be addressed: it seems the new/first searchers do not really buy any performance benefits because it uses so much memory, especially at core loading time. thanks Renee -- http://jetwick.com twitter search prototype
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
just update on this issue... we turned off the new/first searchers (upgrade to Solr 1.4.1), and ran benchmark tests, there is no noticeable performance impact on the queries we perform comparing with Solr 1.3 benchmark tests WITH new/first searchers. Also the memory usage reduced by 5.5 GB after loading the cores with our data volume by turning these static warm caches off. We will take this approach in our production environment but meanwhile I am curious if this issue will be addressed: it seems the new/first searchers do not really buy any performance benefits because it uses so much memory, especially at core loading time. thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1697609.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, I tried the fix suggested in your comments (using solr.TrieDateField ), and it loaded up 130 cores in 1 minute, 1.3GB memory (a little more than 1GB when turning off static warm cache, and much less than 6.5GB when use 'solr.DateField'). Will this have any impact on first query or performance? I am about to run some benchmark test and compare with old data, will update you. Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1637176.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Could Solr just load cores one at a time, waiting for loader events to finish? Or continuously stage 2 or three simultaneously? On Sat, Oct 2, 2010 at 7:08 AM, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun renee_...@mcafee.com wrote: Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. For other's reference: here is the warming query (it's the same for newSearcher too): listener event=firstSearcher class=solr.QuerySenderListener arr name=queries lst str name=qtype:message/str str name=start0/str str name=rows10/str str name=sortmessage_date desc/str /lst /arr /listener The sort field message_date is what will be taking up the memory. Starting with Lucene 2.9 (which is used in Solr 1.4), searching and sorting is per-segment. This is generally beneficial, but in this case I believe it is causing the extra memory usage because the same date value that would have been shared across all documents in the fieldcache is now repeated in each segment it is used in. One potential fix (that requires you to reindex) is to use the date fieldType as defined in the new 1.4 schema: fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ This will use 8 bytes per document in your index, rather than 4 bytes per doc + an array of unique string-date values per index. Trunk (4.0-dev) is also much more efficient at storing string-based fields in the FieldCache - but that will only help you if you're comfortable with using development versions. -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 -- Lance Norskog goks...@gmail.com
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun renee_...@mcafee.com wrote: Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. For other's reference: here is the warming query (it's the same for newSearcher too): listener event=firstSearcher class=solr.QuerySenderListener arr name=queries lst str name=qtype:message/str str name=start0/str str name=rows10/str str name=sortmessage_date desc/str /lst /arr /listener The sort field message_date is what will be taking up the memory. Starting with Lucene 2.9 (which is used in Solr 1.4), searching and sorting is per-segment. This is generally beneficial, but in this case I believe it is causing the extra memory usage because the same date value that would have been shared across all documents in the fieldcache is now repeated in each segment it is used in. One potential fix (that requires you to reindex) is to use the date fieldType as defined in the new 1.4 schema: fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ This will use 8 bytes per document in your index, rather than 4 bytes per doc + an array of unique string-date values per index. Trunk (4.0-dev) is also much more efficient at storing string-based fields in the FieldCache - but that will only help you if you're comfortable with using development versions. -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. I assuem the consequence of commenting out the static warm requests will be it will slow down first time we hit the core for query? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
http://lucene.472066.n3.nabble.com/file/n1617135/solrconfig.xml solrconfig.xml Hi Yonik, I have uploaded our solrconfig.xml file for your reference. we also tried 1.4.1, for same index data, it took about 30-55 minutes to load up all 130 cores, it did not help at all. There is no query running when we tried to upload the cores. Since JConsole is not responding at all when this happens, I am not sure if there is any command link memory profiler I can use to collect information, any suggestions? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617135.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
VERY interesting. Looking forward to what happens on the first queries. Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Fri, 10/1/10, Renee Sun renee_...@mcafee.com wrote: From: Renee Sun renee_...@mcafee.com Subject: Re: Upgrade to Solr 1.4, very slow at start up when loading all cores To: solr-user@lucene.apache.org Date: Friday, October 1, 2010, 2:42 PM Hi Yonik, I attached the solrconfig.xml to you in previous post, and we do have firstSearch and newSearch hook ups. I commented them out, all 130 cores loaded up in 1 minute, same as in solr 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took about 6.5GB for same amount of data. I assuem the consequence of commenting out the static warm requests will be it will slow down first time we hit the core for query? thanks Renee -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1617263.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Thu, Sep 30, 2010 at 10:41 AM, Renee Sun renee_...@mcafee.com wrote: Hi - I posted this problem but no response, I guess I need to post this in the Solr-User forum. Hopefully you will help me on this. We were running Solr 1.3 for long time, with 130 cores. Just upgrade to Solr 1.4, then when we start the Solr, it took about 45 minutes. The catalina.log shows Solr is very slowly loading all the cores. Have you tried 1.4.1 yet? Could you open a JIRA issue for this and give whatever info you can? Info like: - do you have any warming queries configured? - do the cores have documents already, and if so, how many per core? - are you using the same schema solrconfig, or did you upgrade? - have you tried finding out what is taking up all the memory (or all the CPU time)? -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
Hi Yonik, thanks for your reply. I entered a bug for this at : https://issues.apache.org/jira/browse/SOLR-2138 to answer your questions here: - do you have any warming queries configured? no, all autowarmingcount are set to 0 for all caches - do the cores have documents already, and if so, how many per core? yes, 130 cores total, 2,3 of them already have 1~2.4 million documents, others have about 50,000 documents - are you using the same schema solrconfig, or did you upgrade? yes, absolutely no change - have you tried finding out what is taking up all the memory (or all the CPU time)? yes, JConsole shows after 70 cores are loaded in about 4 minutes, all 16GB memory are taken and rest of cores load extremely slow. The memory remain high and never dropped. We are in process to upgrade to 1.4.1 -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-to-Solr-1-4-very-slow-at-start-up-when-loading-all-cores-tp1608728p1611030.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade to solr 1.4
Even i want to upgrade from v1.3 to 1.4 I did 1.3 index directory replace with 1.4 and associated schema changes in that. Its throwing lot of exception like datatype mismatch with Integer, String, Date, etc. Even the results are coming with some error example: str name=AliasERROR:SCHEMA-INDEX-MISMATCH,stringValue=14903346/str Is there any tool/notes to upgrade from 1.3 to 1.4? on Data and schema data types etc? Please suggest us. -Kalidoss.m, Walter Underwood wrote: We are using the script replication. I have no interest in spending time configuring and QA'ing a different method when the scripts work fine. We are running the nightly from 2009-05-11. wunder On 6/26/09 8:51 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Fri, Jun 26, 2009 at 9:11 PM, Walter Underwood wunderw...@netflix.comwrote: Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. Wow, that is good news! Are you also using the java based replication? We deployed 1.4 to all our servers yesterday. Can you tell us which revision you used? Get your world in your inbox! Mail, widgets, documents, spreadsheets, organizer and much more with your Sifymail WIYI id! Log on to http://www.sify.com ** DISCLAIMER ** Information contained and transmitted by this E-MAIL is proprietary to Sify Limited and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If this is a forwarded message, the content of this E-MAIL may not have been sent with the authority of the Company. If you are not the intended recipient, an agent of the intended recipient or a person responsible for delivering the information to the named recipient, you are notified that any use, distribution, transmission, printing, copying or dissemination of this information in any way or in any manner is strictly prohibited. If you have received this communication in error, please delete this mail notify us immediately at ad...@sifycorp.com
Re: Upgrade to solr 1.4
In version 1.3 EventDate field type is date, In 1.4 also its date But we are getting the following error. str name=EventDateERROR:SCHEMA-INDEX-MISMATCH,stringValue=2008-05-16T07:19:28/str -kalidoss.m, kalidoss wrote: Even i want to upgrade from v1.3 to 1.4 I did 1.3 index directory replace with 1.4 and associated schema changes in that. Its throwing lot of exception like datatype mismatch with Integer, String, Date, etc. Even the results are coming with some error example: str name=AliasERROR:SCHEMA-INDEX-MISMATCH,stringValue=14903346/str Is there any tool/notes to upgrade from 1.3 to 1.4? on Data and schema data types etc? Please suggest us. -Kalidoss.m, Walter Underwood wrote: We are using the script replication. I have no interest in spending time configuring and QA'ing a different method when the scripts work fine. We are running the nightly from 2009-05-11. wunder On 6/26/09 8:51 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Fri, Jun 26, 2009 at 9:11 PM, Walter Underwood wunderw...@netflix.comwrote: Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. Wow, that is good news! Are you also using the java based replication? We deployed 1.4 to all our servers yesterday. Can you tell us which revision you used? Get your world in your inbox! Mail, widgets, documents, spreadsheets, organizer and much more with your Sifymail WIYI id! Log on to http://www.sify.com ** DISCLAIMER ** Information contained and transmitted by this E-MAIL is proprietary to Sify Limited and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If this is a forwarded message, the content of this E-MAIL may not have been sent with the authority of the Company. If you are not the intended recipient, an agent of the intended recipient or a person responsible for delivering the information to the named recipient, you are notified that any use, distribution, transmission, printing, copying or dissemination of this information in any way or in any manner is strictly prohibited. If you have received this communication in error, please delete this mail notify us immediately at ad...@sifycorp.com
Re: Upgrade to solr 1.4
On Fri, Nov 20, 2009 at 10:26 AM, kalidoss kalidoss.muthuramalin...@sifycorp.com wrote: In version 1.3 EventDate field type is date, In 1.4 also its date But we are getting the following error. Use the schema you had with 1.3 and it should work. The example schemas are not backward compatible with an index built with the previous version. date in the 1.3 example schema mapped to DateField, but TrieDateField in 1.4. -Yonik http://www.lucidimagination.com
Re: Upgrade to solr 1.4
David Baker wrote: Hi, I need to upgrade from solr 1.3 to solr 1.4. I was wondering if there is a particular revision of 1.4 that I should use that is considered very stable for a production environment? Well it it's not pronounced stable and given in download page I don't think you can rely on being very stable for production environment.
Re: Upgrade to solr 1.4
Solr in general is fairly stable in trunk. That isn't to say that a critical error can't get through, because that does happen, but the test suite is pretty comprehensive. With Solr 1.4 getting closer and closer, I think you'll see the pace of change dropping off. I think it's one of those things that you have to judge for yourself.. Are the features/fixes/enhancements in 1.4 trunk worth a potential risk? I assume that as part of deployment into production you have some sort of defined criteria that says Solr can be added? Testing of server capacity/performance etc? Those might tell you if there are any issues with Solr 1.4 trunk that would need to delay your deployment. Eric On Jun 26, 2009, at 10:58 AM, Julian Davchev wrote: David Baker wrote: Hi, I need to upgrade from solr 1.3 to solr 1.4. I was wondering if there is a particular revision of 1.4 that I should use that is considered very stable for a production environment? Well it it's not pronounced stable and given in download page I don't think you can rely on being very stable for production environment. - Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com Free/Busy: http://tinyurl.com/eric-cal
Re: Upgrade to solr 1.4
Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. We deployed 1.4 to all our servers yesterday. wunder On 6/26/09 7:58 AM, Julian Davchev j...@drun.net wrote: David Baker wrote: Hi, I need to upgrade from solr 1.3 to solr 1.4. I was wondering if there is a particular revision of 1.4 that I should use that is considered very stable for a production environment? Well it it's not pronounced stable and given in download page I don't think you can rely on being very stable for production environment.
Re: Upgrade to solr 1.4
On Fri, Jun 26, 2009 at 9:11 PM, Walter Underwood wunderw...@netflix.comwrote: Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. Wow, that is good news! Are you also using the java based replication? We deployed 1.4 to all our servers yesterday. Can you tell us which revision you used? -- Regards, Shalin Shekhar Mangar.
Re: Upgrade to solr 1.4
We are using the script replication. I have no interest in spending time configuring and QA'ing a different method when the scripts work fine. We are running the nightly from 2009-05-11. wunder On 6/26/09 8:51 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Fri, Jun 26, 2009 at 9:11 PM, Walter Underwood wunderw...@netflix.comwrote: Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. Wow, that is good news! Are you also using the java based replication? We deployed 1.4 to all our servers yesterday. Can you tell us which revision you used?
Re: Upgrade to solr 1.4
We are using a trunk build from approximately the same time with little to no issues including the new replication. -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 From: Shalin Shekhar Mangar shalinman...@gmail.com Reply-To: solr-user@lucene.apache.org Date: Fri, 26 Jun 2009 21:21:44 +0530 To: solr-user@lucene.apache.org Subject: Re: Upgrade to solr 1.4 On Fri, Jun 26, 2009 at 9:11 PM, Walter Underwood wunderw...@netflix.comwrote: Netflix is running a nightly build from May in production. We did our normal QA on it, then ran it on one of our five servers for two weeks. No problems. It is handling about 10% more traffic with 10% less CPU. Wow, that is good news! Are you also using the java based replication? We deployed 1.4 to all our servers yesterday. Can you tell us which revision you used? -- Regards, Shalin Shekhar Mangar.
