On Mon, Feb 22, 2016 at 10:22 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > Highest count is fairly equal between string and text. They are not indexed > but stored and no docvalues used
Ah, that's a big clue... I wonder if it's related to stored fields compression. In Solr 5.5. there is a way to tune compression, but the default is already BEST_SPEED. I don't know if an easy way to turn compression off to see if that is the culprit. Whatever the reasons, this seems to be a large performance regression, and it's not clear if it's expected or not. Could you file a JIRA issue for this? -Yonik > ________________________________________ > From: Yonik Seeley [ysee...@gmail.com] > Sent: 22 February 2016 14:40 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > > What are the types of the fields with the highest count? I assume > they are indexed. Are they stored, and do they have docValues? > > -Yonik > > > On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] <an...@mass.co.uk> > wrote: >> Well I got the numbers wrong, there are actually around 66000 fields on the >> index. I have restructured the index and there are now around 1500 fiields. >> This has resulted in the commit taking 34 seconds which is acceptable for my >> usage however it is still significantly slower than the 4.10.2 commit on the >> original 66000 fields which was around 1 second. >> ________________________________________ >> From: Adam Neal [Extranet] [an...@mass.co.uk] >> Sent: 19 February 2016 17:43 >> To: solr-user@lucene.apache.org >> Subject: RE: Slow commits >> >> I'm out of the office now so I don't have the numbers to hand but from >> memory I think there are probably around 800-1000 fields or so. I will >> confirm on Monday. >> >> If i have time over the weekend I will try and recreate the problem at home >> and see if I can post up a sample. >> ________________________________________ >> From: Yonik Seeley [ysee...@gmail.com] >> Sent: 19 February 2016 16:25 >> To: solr-user@lucene.apache.org >> Subject: Re: Slow commits >> >> On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> >> wrote: >>> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >>> commits on one of my cores. The core in question is relatively small (56k >>> docs) and the issue only shows when commiting after a number of deletes, >>> commiting after additions is fine. As an example commiting after deleting >>> approximately 10% of the documents takes around 25mins. The same test on >>> the 4.10.2 instance takes around 1 second. >>> >>> I have done some investigation and the problem appears to be caused by >>> having dynamic fields, the core in question has a large number, performing >>> the same operation on this core with the dynamic fields removed sees a big >>> improvement on the performance with the commit taking 11 seconds (still not >>> quite on a par with 4.10.2). >> >> Dynamic fields is a Solr schema concept, and does not translate to any >> differences in Lucene. >> You may be hitting something due to a large number of fields (at the >> lucene level, each field name is a different field). How many >> different fields (i.e. fieldnames) do you have across the entire >> index? >> >> -Yonik >> >> ##################################################################################### >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential and >> intended only for the use of the addressee or with its permission. Use by >> anyone else for any purpose is prohibited. If you are not the addressee, >> you should not use, disclose, copy or distribute this e-mail and should >> notify us of receipt immediately by return e-mail to the address where the >> e-mail originated. >> >> This E-mail may not have been sent through a secure system and accordingly >> (i) its contents should not be relied upon by any person without independent >> verification from Mass Consultants Ltd and (ii) it is the responsibility of >> the recipient to ensure that the onward transmission, opening or use of this >> message and any attachments will not adversely affect its systems or data. >> No responsibility is accepted by Mass Consultants Ltd in this regard. >> >> Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be >> monitored by systems or persons other than the addressee, for the purposes >> of ascertaining whether the communication complies with the law and Mass >> Consultants Ltd's policies. >> >> Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, >> Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 >> (0) 1480 222600. >> >> ##################################################################################### >> >> ##################################################################################### >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential and >> intended only for the use of the addressee or with its permission. Use by >> anyone else for any purpose is prohibited. If you are not the addressee, >> you should not use, disclose, copy or distribute this e-mail and should >> notify us of receipt immediately by return e-mail to the address where the >> e-mail originated. >> >> This E-mail may not have been sent through a secure system and accordingly >> (i) its contents should not be relied upon by any person without independent >> verification from Mass Consultants Ltd and (ii) it is the responsibility of >> the recipient to ensure that the onward transmission, opening or use of this >> message and any attachments will not adversely affect its systems or data. >> No responsibility is accepted by Mass Consultants Ltd in this regard. >> >> Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be >> monitored by systems or persons other than the addressee, for the purposes >> of ascertaining whether the communication complies with the law and Mass >> Consultants Ltd's policies. >> >> Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, >> Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 >> (0) 1480 222600. >> >> ##################################################################################### > > ##################################################################################### > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. > > This E-mail may not have been sent through a secure system and accordingly > (i) its contents should not be relied upon by any person without independent > verification from Mass Consultants Ltd and (ii) it is the responsibility of > the recipient to ensure that the onward transmission, opening or use of this > message and any attachments will not adversely affect its systems or data. No > responsibility is accepted by Mass Consultants Ltd in this regard. > > Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be > monitored by systems or persons other than the addressee, for the purposes of > ascertaining whether the communication complies with the law and Mass > Consultants Ltd's policies. > > Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, > Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 > (0) 1480 222600. > > #####################################################################################
