What are the types of the fields with the highest count? I assume they are indexed. Are they stored, and do they have docValues?
-Yonik On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > Well I got the numbers wrong, there are actually around 66000 fields on the > index. I have restructured the index and there are now around 1500 fiields. > This has resulted in the commit taking 34 seconds which is acceptable for my > usage however it is still significantly slower than the 4.10.2 commit on the > original 66000 fields which was around 1 second. > ________________________________________ > From: Adam Neal [Extranet] [an...@mass.co.uk] > Sent: 19 February 2016 17:43 > To: solr-user@lucene.apache.org > Subject: RE: Slow commits > > I'm out of the office now so I don't have the numbers to hand but from memory > I think there are probably around 800-1000 fields or so. I will confirm on > Monday. > > If i have time over the weekend I will try and recreate the problem at home > and see if I can post up a sample. > ________________________________________ > From: Yonik Seeley [ysee...@gmail.com] > Sent: 19 February 2016 16:25 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> > wrote: >> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >> commits on one of my cores. The core in question is relatively small (56k >> docs) and the issue only shows when commiting after a number of deletes, >> commiting after additions is fine. As an example commiting after deleting >> approximately 10% of the documents takes around 25mins. The same test on the >> 4.10.2 instance takes around 1 second. >> >> I have done some investigation and the problem appears to be caused by >> having dynamic fields, the core in question has a large number, performing >> the same operation on this core with the dynamic fields removed sees a big >> improvement on the performance with the commit taking 11 seconds (still not >> quite on a par with 4.10.2). > > Dynamic fields is a Solr schema concept, and does not translate to any > differences in Lucene. > You may be hitting something due to a large number of fields (at the > lucene level, each field name is a different field). How many > different fields (i.e. fieldnames) do you have across the entire > index? > > -Yonik > > ##################################################################################### > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. > > This E-mail may not have been sent through a secure system and accordingly > (i) its contents should not be relied upon by any person without independent > verification from Mass Consultants Ltd and (ii) it is the responsibility of > the recipient to ensure that the onward transmission, opening or use of this > message and any attachments will not adversely affect its systems or data. No > responsibility is accepted by Mass Consultants Ltd in this regard. > > Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be > monitored by systems or persons other than the addressee, for the purposes of > ascertaining whether the communication complies with the law and Mass > Consultants Ltd's policies. > > Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, > Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 > (0) 1480 222600. > > ##################################################################################### > > ##################################################################################### > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. > > This E-mail may not have been sent through a secure system and accordingly > (i) its contents should not be relied upon by any person without independent > verification from Mass Consultants Ltd and (ii) it is the responsibility of > the recipient to ensure that the onward transmission, opening or use of this > message and any attachments will not adversely affect its systems or data. No > responsibility is accepted by Mass Consultants Ltd in this regard. > > Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be > monitored by systems or persons other than the addressee, for the purposes of > ascertaining whether the communication complies with the law and Mass > Consultants Ltd's policies. > > Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, > Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 > (0) 1480 222600. > > #####################################################################################
