RE: Slow commits
I have also tried it with the fields just indexed and not stored, the performance is the same so I doubt it is related to stored field compression. I can file a JIRA, unfortunately I wont be able to provide example files for this system but I will try and reproduce it with some test data and include that. From: Yonik Seeley [ysee...@gmail.com] Sent: 22 February 2016 15:43 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Mon, Feb 22, 2016 at 10:22 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > Highest count is fairly equal between string and text. They are not indexed > but stored and no docvalues used Ah, that's a big clue... I wonder if it's related to stored fields compression. In Solr 5.5. there is a way to tune compression, but the default is already BEST_SPEED. I don't know if an easy way to turn compression off to see if that is the culprit. Whatever the reasons, this seems to be a large performance regression, and it's not clear if it's expected or not. Could you file a JIRA issue for this? -Yonik > > From: Yonik Seeley [ysee...@gmail.com] > Sent: 22 February 2016 14:40 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > > What are the types of the fields with the highest count? I assume > they are indexed. Are they stored, and do they have docValues? > > -Yonik > > > On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] <an...@mass.co.uk> > wrote: >> Well I got the numbers wrong, there are actually around 66000 fields on the >> index. I have restructured the index and there are now around 1500 fiields. >> This has resulted in the commit taking 34 seconds which is acceptable for my >> usage however it is still significantly slower than the 4.10.2 commit on the >> original 66000 fields which was around 1 second. >> >> From: Adam Neal [Extranet] [an...@mass.co.uk] >> Sent: 19 February 2016 17:43 >> To: solr-user@lucene.apache.org >> Subject: RE: Slow commits >> >> I'm out of the office now so I don't have the numbers to hand but from >> memory I think there are probably around 800-1000 fields or so. I will >> confirm on Monday. >> >> If i have time over the weekend I will try and recreate the problem at home >> and see if I can post up a sample. >> >> From: Yonik Seeley [ysee...@gmail.com] >> Sent: 19 February 2016 16:25 >> To: solr-user@lucene.apache.org >> Subject: Re: Slow commits >> >> On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> >> wrote: >>> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >>> commits on one of my cores. The core in question is relatively small (56k >>> docs) and the issue only shows when commiting after a number of deletes, >>> commiting after additions is fine. As an example commiting after deleting >>> approximately 10% of the documents takes around 25mins. The same test on >>> the 4.10.2 instance takes around 1 second. >>> >>> I have done some investigation and the problem appears to be caused by >>> having dynamic fields, the core in question has a large number, performing >>> the same operation on this core with the dynamic fields removed sees a big >>> improvement on the performance with the commit taking 11 seconds (still not >>> quite on a par with 4.10.2). >> >> Dynamic fields is a Solr schema concept, and does not translate to any >> differences in Lucene. >> You may be hitting something due to a large number of fields (at the >> lucene level, each field name is a different field). How many >> different fields (i.e. fieldnames) do you have across the entire >> index? >> >> -Yonik >> >> # >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential and >> intended only for the use of the addressee or with its permission. Use by >> anyone else for any purpose is prohibited. If you are not the addressee, >> you should not use, disclose, copy or distribute this e-mail and should >> notify us of receipt immediately by return e-mail to the address where the >> e-mail originated. >> >> This E-mail may not have been sent through a secure system and accordingly >> (i) its contents should not be relied upon by any person without independent >> verification from Mass Consultants Ltd and (ii) it is the responsibility of >> th
RE: Slow commits
Highest count is fairly equal between string and text. They are not indexed but stored and no docvalues used From: Yonik Seeley [ysee...@gmail.com] Sent: 22 February 2016 14:40 To: solr-user@lucene.apache.org Subject: Re: Slow commits What are the types of the fields with the highest count? I assume they are indexed. Are they stored, and do they have docValues? -Yonik On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > Well I got the numbers wrong, there are actually around 66000 fields on the > index. I have restructured the index and there are now around 1500 fiields. > This has resulted in the commit taking 34 seconds which is acceptable for my > usage however it is still significantly slower than the 4.10.2 commit on the > original 66000 fields which was around 1 second. > ____ > From: Adam Neal [Extranet] [an...@mass.co.uk] > Sent: 19 February 2016 17:43 > To: solr-user@lucene.apache.org > Subject: RE: Slow commits > > I'm out of the office now so I don't have the numbers to hand but from memory > I think there are probably around 800-1000 fields or so. I will confirm on > Monday. > > If i have time over the weekend I will try and recreate the problem at home > and see if I can post up a sample. > > From: Yonik Seeley [ysee...@gmail.com] > Sent: 19 February 2016 16:25 > To: solr-user@lucene.apache.org > Subject: Re: Slow commits > > On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> > wrote: >> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow >> commits on one of my cores. The core in question is relatively small (56k >> docs) and the issue only shows when commiting after a number of deletes, >> commiting after additions is fine. As an example commiting after deleting >> approximately 10% of the documents takes around 25mins. The same test on the >> 4.10.2 instance takes around 1 second. >> >> I have done some investigation and the problem appears to be caused by >> having dynamic fields, the core in question has a large number, performing >> the same operation on this core with the dynamic fields removed sees a big >> improvement on the performance with the commit taking 11 seconds (still not >> quite on a par with 4.10.2). > > Dynamic fields is a Solr schema concept, and does not translate to any > differences in Lucene. > You may be hitting something due to a large number of fields (at the > lucene level, each field name is a different field). How many > different fields (i.e. fieldnames) do you have across the entire > index? > > -Yonik > > # > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail > originated. > > This E-mail may not have been sent through a secure system and accordingly > (i) its contents should not be relied upon by any person without independent > verification from Mass Consultants Ltd and (ii) it is the responsibility of > the recipient to ensure that the onward transmission, opening or use of this > message and any attachments will not adversely affect its systems or data. No > responsibility is accepted by Mass Consultants Ltd in this regard. > > Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be > monitored by systems or persons other than the addressee, for the purposes of > ascertaining whether the communication complies with the law and Mass > Consultants Ltd's policies. > > Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, > Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 > (0) 1480 222600. > > # > > # > > This E-mail is the property of Mass Consultants Ltd. It is confidential and > intended only for the use of the addressee or with its permission. Use by > anyone else for any purpose is prohibited. If you are not the addressee, you > should not use, disclose, copy or distribute this e-mail and should notify us > of receipt immediately by return e-mail to the address where the e-mail >
RE: Slow commits
Yup, that's correct. Not talking massive amounts of data really. The commit performance difference between 4.10.2 and 5.3.1 is huge in this case. From: Susheel Kumar [susheel2...@gmail.com] Sent: 22 February 2016 13:31 To: solr-user@lucene.apache.org Subject: Re: Slow commits Sorry, I see now you mentioned 56K docs which is pretty small. On Mon, Feb 22, 2016 at 8:30 AM, Susheel Kumar <susheel2...@gmail.com> wrote: > Adam - how many documents you have in your index? > > Thanks, > Susheel > > On Mon, Feb 22, 2016 at 4:37 AM, Adam Neal [Extranet] <an...@mass.co.uk> > wrote: > >> Well I got the numbers wrong, there are actually around 66000 fields on >> the index. I have restructured the index and there are now around 1500 >> fiields. This has resulted in the commit taking 34 seconds which is >> acceptable for my usage however it is still significantly slower than the >> 4.10.2 commit on the original 66000 fields which was around 1 second. >> >> From: Adam Neal [Extranet] [an...@mass.co.uk] >> Sent: 19 February 2016 17:43 >> To: solr-user@lucene.apache.org >> Subject: RE: Slow commits >> >> I'm out of the office now so I don't have the numbers to hand but from >> memory I think there are probably around 800-1000 fields or so. I will >> confirm on Monday. >> >> If i have time over the weekend I will try and recreate the problem at >> home and see if I can post up a sample. >> >> From: Yonik Seeley [ysee...@gmail.com] >> Sent: 19 February 2016 16:25 >> To: solr-user@lucene.apache.org >> Subject: Re: Slow commits >> >> On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> >> wrote: >> > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with >> slow commits on one of my cores. The core in question is relatively small >> (56k docs) and the issue only shows when commiting after a number of >> deletes, commiting after additions is fine. As an example commiting after >> deleting approximately 10% of the documents takes around 25mins. The same >> test on the 4.10.2 instance takes around 1 second. >> > >> > I have done some investigation and the problem appears to be caused by >> having dynamic fields, the core in question has a large number, performing >> the same operation on this core with the dynamic fields removed sees a big >> improvement on the performance with the commit taking 11 seconds (still not >> quite on a par with 4.10.2). >> >> Dynamic fields is a Solr schema concept, and does not translate to any >> differences in Lucene. >> You may be hitting something due to a large number of fields (at the >> lucene level, each field name is a different field). How many >> different fields (i.e. fieldnames) do you have across the entire >> index? >> >> -Yonik >> >> >> # >> >> This E-mail is the property of Mass Consultants Ltd. It is confidential >> and intended only for the use of the addressee or with its permission. Use >> by anyone else for any purpose is prohibited. If you are not the >> addressee, you should not use, disclose, copy or distribute this e-mail and >> should notify us of receipt immediately by return e-mail to the address >> where the e-mail originated. >> >> This E-mail may not have been sent through a secure system and >> accordingly (i) its contents should not be relied upon by any person >> without independent verification from Mass Consultants Ltd and (ii) it is >> the responsibility of the recipient to ensure that the onward transmission, >> opening or use of this message and any attachments will not adversely >> affect its systems or data. No responsibility is accepted by Mass >> Consultants Ltd in this regard. >> >> Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may >> be monitored by systems or persons other than the addressee, for the >> purposes of ascertaining whether the communication complies with the law >> and Mass Consultants Ltd's policies. >> >> Mass Consultants Ltd is registered in England No. 1705804, Enterprise >> House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. >> Tel: +44 (0) 1480 222600. >> >> >> # >> >> >> #
RE: Slow commits
Well I got the numbers wrong, there are actually around 66000 fields on the index. I have restructured the index and there are now around 1500 fiields. This has resulted in the commit taking 34 seconds which is acceptable for my usage however it is still significantly slower than the 4.10.2 commit on the original 66000 fields which was around 1 second. From: Adam Neal [Extranet] [an...@mass.co.uk] Sent: 19 February 2016 17:43 To: solr-user@lucene.apache.org Subject: RE: Slow commits I'm out of the office now so I don't have the numbers to hand but from memory I think there are probably around 800-1000 fields or so. I will confirm on Monday. If i have time over the weekend I will try and recreate the problem at home and see if I can post up a sample. From: Yonik Seeley [ysee...@gmail.com] Sent: 19 February 2016 16:25 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. > > I have done some investigation and the problem appears to be caused by having > dynamic fields, the core in question has a large number, performing the same > operation on this core with the dynamic fields removed sees a big improvement > on the performance with the commit taking 11 seconds (still not quite on a > par with 4.10.2). Dynamic fields is a Solr schema concept, and does not translate to any differences in Lucene. You may be hitting something due to a large number of fields (at the lucene level, each field name is a different field). How many different fields (i.e. fieldnames) do you have across the entire index? -Yonik # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. # # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
I'm out of the office now so I don't have the numbers to hand but from memory I think there are probably around 800-1000 fields or so. I will confirm on Monday. If i have time over the weekend I will try and recreate the problem at home and see if I can post up a sample. From: Yonik Seeley [ysee...@gmail.com] Sent: 19 February 2016 16:25 To: solr-user@lucene.apache.org Subject: Re: Slow commits On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <an...@mass.co.uk> wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. > > I have done some investigation and the problem appears to be caused by having > dynamic fields, the core in question has a large number, performing the same > operation on this core with the dynamic fields removed sees a big improvement > on the performance with the commit taking 11 seconds (still not quite on a > par with 4.10.2). Dynamic fields is a Solr schema concept, and does not translate to any differences in Lucene. You may be hitting something due to a large number of fields (at the lucene level, each field name is a different field). How many different fields (i.e. fieldnames) do you have across the entire index? -Yonik # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
max heap on both instances is the same at 8gig, and only using around 1.5gig at the time of testing. 5.3.1 index size is 90MB 4.10.2 index size is 125MB From: Shawn Heisey [apa...@elyograg.org] Sent: 19 February 2016 15:45 To: solr-user@lucene.apache.org Subject: Re: Slow commits On 2/19/2016 6:51 AM, Adam Neal [Extranet] wrote: > I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow > commits on one of my cores. The core in question is relatively small (56k > docs) and the issue only shows when commiting after a number of deletes, > commiting after additions is fine. As an example commiting after deleting > approximately 10% of the documents takes around 25mins. The same test on the > 4.10.2 instance takes around 1 second. This sounds like one of two possibilities, with a strong possibility that it's both: * You are encountering extreme garbage collection pauses from a heap that's too small. * The deletes are leading to segment merges that are proceeding slowly. What is the max heap set to on the new version? Do you know what it was set to on the old version? How much disk space is used by your 56000 document index? Thanks, Shawn # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
RE: Slow commits
Just some additional information, the problem is mainly when the dynamic fields are stored. Just having them indexed reduces the commit time to around 20 seconds. Unfortunately I need them stored. From: Adam Neal [Extranet] [an...@mass.co.uk] Sent: 19 February 2016 13:51 To: solr-user@lucene.apache.org Subject: Slow commits I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow commits on one of my cores. The core in question is relatively small (56k docs) and the issue only shows when commiting after a number of deletes, commiting after additions is fine. As an example commiting after deleting approximately 10% of the documents takes around 25mins. The same test on the 4.10.2 instance takes around 1 second. I have done some investigation and the problem appears to be caused by having dynamic fields, the core in question has a large number, performing the same operation on this core with the dynamic fields removed sees a big improvement on the performance with the commit taking 11 seconds (still not quite on a par with 4.10.2). At the moment for this core it is quicker in 5.3.1 to delete everything and reindex the data than it is to delete old documents. Are there any changes that may have caused this or anything I should be doing differently in 5.3.1? I intend to download 5.4.1 next week to see if that improves things. Thanks Adam # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
Slow commits
I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow commits on one of my cores. The core in question is relatively small (56k docs) and the issue only shows when commiting after a number of deletes, commiting after additions is fine. As an example commiting after deleting approximately 10% of the documents takes around 25mins. The same test on the 4.10.2 instance takes around 1 second. I have done some investigation and the problem appears to be caused by having dynamic fields, the core in question has a large number, performing the same operation on this core with the dynamic fields removed sees a big improvement on the performance with the commit taking 11 seconds (still not quite on a par with 4.10.2). At the moment for this core it is quicker in 5.3.1 to delete everything and reindex the data than it is to delete old documents. Are there any changes that may have caused this or anything I should be doing differently in 5.3.1? I intend to download 5.4.1 next week to see if that improves things. Thanks Adam # This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only for the use of the addressee or with its permission. Use by anyone else for any purpose is prohibited. If you are not the addressee, you should not use, disclose, copy or distribute this e-mail and should notify us of receipt immediately by return e-mail to the address where the e-mail originated. This E-mail may not have been sent through a secure system and accordingly (i) its contents should not be relied upon by any person without independent verification from Mass Consultants Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by Mass Consultants Ltd in this regard. Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems or persons other than the addressee, for the purposes of ascertaining whether the communication complies with the law and Mass Consultants Ltd's policies. Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road, Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600. #
