RE: Potential Slow searching for unified highlighting on Solr 8.8.0/8.8.1
Hi Ere
Please to be of service!
No I have not filed a JIRA ticket. I am new to interacting with the Solr
Community and only beginning to 'find my legs'. I am not too sure what JIRA
is I am afraid!
Regards
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
-Original Message-
From: Ere Maijala
Sent: 01 March 2021 12:53
To: solr-user@lucene.apache.org
Subject: Re: Potential Slow searching for unified highlighting on Solr
8.8.0/8.8.1
EXTERNAL EMAIL - Be cautious of all links and attachments.
Hi,
Whoa, thanks for the heads-up! You may just have saved me from a whole lot
of trouble. Did you file a JIRA ticket already?
Thanks,
Ere
Flowerday, Matthew J kirjoitti 1.3.2021 klo 14.00:
> Hi There
>
> I just came across a situation where a unified highlighting search
> under solr 8.8.0/8.8.1 can take over 20 mins to run and eventually times
out.
> I resolved it by a config change but it can catch you out. Hence
> this email.
>
> With solr 8.8.0 a new unified highlighting parameter
> was implemented which if not set defaults to 0.5.
> This attempts to improve the high lighting so that highlighted text
> does not appear right at the left. This works well but if you have a
> search result with numerous occurrences of the word in question within
> the record performance goes right down!
>
> 2021-02-27 06:45:03.151 INFO (qtp762476028-20) [ x:uleaf]
> o.a.s.c.S.Request [uleaf] webapp=/solr path=/select
> params={hl.snippets=2=test=on=100=id,d
> escription,specification,score=20=*=10&_=161440511913
> 4}
> hits=57008 status=0 QTime=1414320
>
> 2021-02-27 06:45:03.245 INFO (qtp762476028-20) [ x:uleaf]
> o.a.s.s.HttpSolrCall Unable to write response, client closed
> connection or we are shutting down =>
> org.eclipse.jetty.io.EofException
>
>at
> org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
>
> org.eclipse.jetty.io.EofException: null
>
>at
> org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
>at
> org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
>at
> org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378)
> ~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
>
> when I set =0.25 results came back much quicker
>
> 2021-02-27 14:59:57.189 INFO (qtp1291367132-24) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.25=100=2=test
> axAnalyzedChars=100=*=unified=9&_=
> 1614430061690}
> hits=136939 status=0 QTime=87024
>
> And =0.1
>
> 2021-02-27 15:18:45.542 INFO (qtp1291367132-19) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.1=100=2=test
> xAnalyzedChars=100=*=unified=9&_=1
> 614430061690}
> hits=136939 status=0 QTime=69033
>
> And =0.0
>
> 2021-02-27 15:20:38.194 INFO (qtp1291367132-24) [ x:holmes]
> o.a.s.c.S.Request [holmes] webapp=/solr path=/select
> params={hl.weightMatches=false=on=id,description,specification,s
> core=1=0.0=100=2=test
> xAnalyzedChars=100=*=unified=9&_=1
> 614430061690}
> hits=136939 status=0 QTime=2841
>
> I left our setting at 0.0 this presumably how it was in 7.7.1 (fully
> left aligned). I am not too sure as to how many time a word has to
> occur in a record for performance to go right down but if too many
> it can have a BIG impact.
>
> I also noticed that setting =9 did not break out of
> the query until it finished. Perhaps because the query finished
> quickly and what took the time was the highlighting. It might be an
> idea to get to also cover any highlighting so that the
> query does not run until the jetty timeout is hit. The machine 100%
> one core for about
> 20 mins!.
>
> Hope this helps.
>
> Regards
>
> Matthew
>
> *Matthew Flowerday*| Consultant | ULEAF
>
> Unisys | 01908 774830| matthew.flower...@unisys.com
> <mailto:matthew.flower...@unisys.com>
>
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes |
> MK17 8LX
>
> unisys_logo <http://www.
Potential Slow searching for unified highlighting on Solr 8.8.0/8.8.1
Hi There
I just came across a situation where a unified highlighting search under
solr 8.8.0/8.8.1 can take over 20 mins to run and eventually times out. I
resolved it by a config change - but it can catch you out. Hence this email.
With solr 8.8.0 a new unified highlighting parameter was
implemented which if not set defaults to 0.5. This attempts to improve the
high lighting so that highlighted text does not appear right at the left.
This works well but if you have a search result with numerous occurrences of
the word in question within the record performance goes right down!
2021-02-27 06:45:03.151 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.c.S.Request [uleaf] webapp=/solr path=/select
params={hl.snippets=2=test=on=100=id,descrip
tion,specification,score=20=*=10&_=1614405119134}
hits=57008 status=0 QTime=1414320
2021-02-27 06:45:03.245 INFO (qtp762476028-20) [ x:uleaf]
o.a.s.s.HttpSolrCall Unable to write response, client closed connection or
we are shutting down => org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
org.eclipse.jetty.io.EofException: null
at
org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378)
~[jetty-io-9.4.34.v20201102.jar:9.4.34.v20201102]
when I set =0.25 results came back much quicker
2021-02-27 14:59:57.189 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score
tart=1=0.25=100=2=test
ars=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=87024
And =0.1
2021-02-27 15:18:45.542 INFO (qtp1291367132-19) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score
tart=1=0.1=100=2=test
rs=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=69033
And =0.0
2021-02-27 15:20:38.194 INFO (qtp1291367132-24) [ x:holmes]
o.a.s.c.S.Request [holmes] webapp=/solr path=/select
params={hl.weightMatches=false=on=id,description,specification,score
tart=1=0.0=100=2=test
rs=100=*=unified=9&_=1614430061690}
hits=136939 status=0 QTime=2841
I left our setting at 0.0 - this presumably how it was in 7.7.1 (fully left
aligned). I am not too sure as to how many time a word has to occur in a
record for performance to go right down - but if too many it can have a BIG
impact.
I also noticed that setting =9 did not break out of the
query until it finished. Perhaps because the query finished quickly and what
took the time was the highlighting. It might be an idea to get
to also cover any highlighting so that the query does not run until the
jetty timeout is hit. The machine 100% one core for about 20 mins!.
Hope this helps.
Regards
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830|
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
Unable to connect to an 8.8.0 Solr Cloud database via API
Hi There I have been checking out the latest (8.8.0) SolrCloud database (using Zookeeper 3.6.2) against our application which talks to Solr via the Solr API (I am not too sure of the details as I am not a java developer unfortunately!). The software has Solr 8.7.0/ZooKeeper 3.6.2 libraries 'embedded'. When our application sends a query to the SolrCloud 8.8.0 database it generates this error: 2021-02-05 05:36:42,644 INFO [org.apache.solr.common.cloud.ConnectionManager] (default task-7) Client is connected to ZooKeeper 2021-02-05 05:36:42,685 INFO [org.apache.solr.common.cloud.ZkStateReader] (default task-7) Updated live nodes from ZooKeeper... (0) -> (1) 2021-02-05 05:36:42,752 INFO [org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider] (default task-7) Cluster at localhost:2185 ready 2021-02-05 05:36:42,785 INFO [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7) request was not communication error it seems 2021-02-05 05:36:42,788 INFO [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7) Request to collection [holmes] failed due to (0) java.lang.NullPointerException, retry=0 maxRetries=5 commError=false errorCode=0 2021-02-05 05:36:42,789 ERROR [com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl] (default task-7) Solr exception on query 'test': org.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException at deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient. requestWithRetryOnStaleState(BaseCloudSolrClient.java:1053) at deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient. request(BaseCloudSolrClient.java:866) at deployment.uleaf.ear//org.apache.solr.client.solrj.SolrRequest.process(SolrR equest.java:214) at deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClie nt.java:1003) at deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClie nt.java:1018) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrConnect ionManagerImpl.search(SolrConnectionManagerImpl.java:191) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchS erviceImpl.search(SolrSearchServiceImpl.java:223) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchS erviceImpl.retrieveSearchPages(SolrSearchServiceImpl.java:163) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchS erviceImpl.retrieveResults(SolrSearchServiceImpl.java:147) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchS erviceImpl.search(SolrSearchServiceImpl.java:132) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchS erviceImpl.search(SolrSearchServiceImpl.java:118) at deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr. presentation.SolrController.search(SolrController.java:163) at deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr. presentation.SolrController.searchValid(SolrController.java:127) at deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr. presentation.SolrController.search(SolrController.java:108) at deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr. presentation.SolrController$$FastClassBySpringCGLIB$$82e1da1b.invoke() at deployment.uleaf.ear//org.springframework.cglib.proxy.MethodProxy.invoke(Met hodProxy.java:204) at deployment.uleaf.ear//org.springframework.aop.framework.CglibAopProxy$CglibM ethodInvocation.invokeJoinpoint(CglibAopProxy.java:708) at deployment.uleaf.ear//org.springframework.aop.framework.ReflectiveMethodInvo cation.proceed(ReflectiveMethodInvocation.java:157) at deployment.uleaf.ear//org.springframework.aop.aspectj.MethodInvocationProcee dingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.persistence.support.AbstractRe questLogger.proceedWithoutDebug(AbstractRequestLogger.java:60) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.persistence.support.AbstractRe questLogger.logAndInvoke(AbstractRequestLogger.java:36) at deployment.uleaf.ear//com.unisys.holmes2.h2ng.presentation.support.RequestLo gger.request(RequestLogger.java:48) at jdk.internal.reflect.GeneratedMethodAccessor733.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.uleaf.ear//org.springframework.aop.aspectj.AbstractAspectJAdvice.
RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There Just as an update to this thread I have resolved the issue. The new schema.xml had this entries Once I commented out the lines containing _root_ and _nest_path_ (as we don't have nested documents) and re-started solr then no further duplication on update occurred. Regards Matthew Matthew Flowerday | Consultant | ULEAF Unisys | 01908 774830| matthew.flower...@unisys.com Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices. -Original Message- From: Flowerday, Matthew J Sent: 15 January 2021 11:18 To: solr-user@lucene.apache.org Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0 EXTERNAL EMAIL - Be cautious of all links and attachments. smime.p7s Description: S/MIME cryptographic signature
RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi Jim
Thanks for looking into it for me.
I did some more testing and if I created a base solr 7.7.1 database using
the 'out of the box' schema.xml and solrconfig and add this item manually
using the Solr Admin tool documents/XML
ABCD-N1
A test
And then update it using
ABCD-N1
A test updated
It correctly updates and deletes the old copy.
I then 'migrated' it to solr 8.7.0 and updated the record in the same manner
(using documents/XML) with this
ABCD-N1
A test updated again
It created a new record without deleting the old record
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"*:*",
"_":"1610703647168"}},
"response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[
{
"id":"ABCD-N1",
"title_t":"A test updated",
"_version_":1688944583266795520},
{
"id":"ABCD-N1",
"title_t":"A test updated again",
"_version_":1688950299184594944}]
}}
It is almost as if the delete of the record from the segment set up 7.7.1 is
not recognised.
When I updated the record again using
ABCD-N1
A test updated again and again
It updated the newly created record and deleted the old version.
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"*:*",
"_":"1610703647168"}},
"response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[
{
"id":"ABCD-N1",
"title_t":"A test updated",
"_version_":1688944583266795520},
{
"id":"ABCD-N1",
"title_t":"A test updated again and again",
"_version_":1688950897568120832}]
}}
I did further testing by turning on lucene TRACE on my database and first
update generated
2021-01-15 09:38:30.138 INFO (qtp1458091526-18) [ x:uleaf]
o.a.s.u.LoggingInfoStream [BD][qtp1458091526-18]: now apply del packet
(org.apache.solr.update.SolrIndexWriter@15e9adf2
<mailto:org.apache.solr.update.SolrIndexWriter@15e9adf2> ) to 10 segments,
mergeGen 0
2021-01-15 09:38:30.138 INFO (qtp1458091526-18) [ x:uleaf]
o.a.s.u.LoggingInfoStream [BD][qtp1458091526-18]: applyTermDeletes took 0.44
msec for 10 segments and 1 del terms; 0 new deletions
Whilst the second update generated
2021-01-15 09:44:21.543 INFO (qtp1458091526-17) [ x:uleaf]
o.a.s.u.LoggingInfoStream [BD][qtp1458091526-17]: now apply del packet
(org.apache.solr.update.SolrIndexWriter@15e9adf2
<mailto:org.apache.solr.update.SolrIndexWriter@15e9adf2> ) to 11 segments,
mergeGen 0
2021-01-15 09:44:21.544 INFO (qtp1458091526-17) [ x:uleaf]
o.a.s.u.LoggingInfoStream [BD][qtp1458091526-17]: applyTermDeletes took 0.29
msec for 11 segments and 1 del terms; 1 new deletions
I think that it does not seem to find the document to delete in the old
segment.
Could this be a bug in Solr?
Many thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <mailto:matthew.flower...@unisys.com>
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
<http://www.linkedin.com/company/unisys><http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel>
<http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys>
<http://blogs.unisys.com/>
From: Dyer, Jim
Sent: 13 January 2021 18:21
To: solr-user@lucene.apache.org
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0
EXTERNAL EMAIL - Be cautious of all links and attachments.
I think if you have _root_ in schema.xml you should look elsewhere. My
memory is merely adding this one line to schema.xml took care of our
problem.
From: Flowerday, Matthew J mailto:matthew.flower...@gb.unisys.com> >
Sent: Tuesday, January 12, 2021 3:23 AM
To: solr-user@lucene.apache.org <mailto:solr-user@lucene.apache.org>
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi Jim
Thanks for getting back to me.
I checked the schema.xml that we are using and it has the line you
mentioned:
And this is the only reference (apart from within a comment
RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi Jim
Thanks for getting back to me.
I checked the schema.xml that we are using and it has the line you
mentioned:
And this is the only reference (apart from within a comment) for _root_ In
the schema.xml. Does your schema.xml have further references to _root_ that
I could need? I also checked out solrconfig.xml file for any references to
_root_ and there are none.
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <mailto:matthew.flower...@unisys.com>
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
<http://www.linkedin.com/company/unisys><http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel>
<http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys>
<http://blogs.unisys.com/>
From: Dyer, Jim
Sent: 11 January 2021 22:58
To: solr-user@lucene.apache.org
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0
EXTERNAL EMAIL - Be cautious of all links and attachments.
When we upgraded from 7.x to 8.x, I ran into an issue similar to yours:
when updating an existing document in the index, the document would be
duplicated instead of replaced as expected. The solution was to add a
"_root_" field to schema.xml like this:
It appeared that when a feature was added for nested documents, this field
somehow became mandatory in order for updates to work properly, at least in
some cases.
From: Flowerday, Matthew J mailto:matthew.flower...@gb.unisys.com> >
Sent: Saturday, January 9, 2021 4:44 AM
To: solr-user@lucene.apache.org <mailto:solr-user@lucene.apache.org>
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There
As a test I stopped Solr and ran the IndexUpgrader tool on the database to
see if this might fix the issue. It completed OK but unfortunately the issue
still occurs - a new version of the record on solr is created rather than
updating the original record.
It looks to me as if the record created under 7.7.1 is somehow not being
'marked as deleted' in the way that records created under 8.7.0 are. Is
there a way for these records to be marked as deleted when they are updated.
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <mailto:matthew.flower...@unisys.com>
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
<http://www.linkedin.com/company/unisys><http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel>
<http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys>
<http://blogs.unisys.com/>
From: Flowerday, Matthew J mailto:matthew.flower...@gb.unisys.com> >
Sent: 07 January 2021 12:25
To: solr-user@lucene.apache.org <mailto:solr-user@lucene.apache.org>
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There
I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped
the database and re-indexed (as this would take too long to run on site).
On my local windows machine I have a single solr server 7.7.1 installation
I upgraded in the following manner
* Installed windows solr 8.7.0 on my machine in a different folder
* Copied the core related folder (holding conf, data, lib,
core.properties) from 7.7.1 to the new 8.7.0 folder
* Brought up the solr
* Checked that queries work through the Solr Admin Tool and our
application
This all worked fine until I tried to update a record which had been created
under 7.7.1. Instead of marking the old record as deleted it effectively
created a new copy of the record with the change in and left the old image
as still visible. When I updated the record again it then correctly updated
the new 8.7.0 version without leaving the old image behind. If I created a
new record and then updated it the solr record would be updated correctly.
The issue only seemed to affect the old 7.7.1 created records.
An example of the duplication as follows (the first record is 7.7.1 created
version and the second record is the 8.7.0 version after carrying out an
update):
{
"responseHeader":{
"status":0,
"
RE: Re:Query over migrating a solr database from 7.7.1 to 8.7.0
s={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
:id=dop9d815p1kjrk1phi955lbgl
_1p(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10,
java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0,
os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0,
timestamp=1610097594499}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
:id=dop9d815p1kjrk1phi955lbgt" [6 segments ; isCommit = false]
IFD 0 [2021-01-09T10:16:55.457043500Z; main]: 10 msec to checkpoint
I was not aware about the need to carry out an commit after this. Please can
you show me how to do this. I checked the Index folder before and after running
the tool and the number of files reduced – so it seemed to have done some
‘combining’.
If I run the tool again it seems to run through similar dialog and does not
report that nothing needs correcting.
Also please could you confirm if my steps to upgrade a solr database to a new
release are correct (just in case what I did caused this issue).
* Installed windows solr 8.7.0 on my machine in a different folder
* Copied the core related folder (holding conf, data, lib,
core.properties) from 7.7.1 to the new 8.7.0 folder
* Brought up the solr
* Checked that queries work through the Solr Admin Tool and our
application
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <mailto:matthew.flower...@unisys.com>
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this in
error, please contact the sender and delete the e-mail and its attachments from
all devices.
<http://www.linkedin.com/company/unisys><http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel> <http://www.facebook.com/unisyscorp>
<https://vimeo.com/unisys> <http://blogs.unisys.com/>
From: matthew sporleder
Sent: 10 January 2021 21:38
To: solr-user@lucene.apache.org
Subject: Re: Re:Query over migrating a solr database from 7.7.1 to 8.7.0
EXTERNAL EMAIL - Be cautious of all links and attachments.
I think the general advice is to do a full re-index on a major version upgrade.
Also - did you ever commit?
On Sun, Jan 10, 2021 at 11:13 AM Flowerday, Matthew J
mailto:matthew.flower...@gb.unisys.com> >
wrote:
Hi There
Thanks for contacting me.
I carried out this analysis of the solr log from the updates I carried out at
the time:
Looking at the update requests sent to Solr. The first update of an existing
record generated
2021-01-07 06:04:18.958 INFO (qtp1458091526-17) [ x:uleaf]
o.a.s.u.p.LogUpdateProcessorFactory [uleaf] webapp=/solr path=/update
params={wt=javabin=2}{add=[9901020319M01-X11 (1688206792619720704)]} 0
59
2021-01-07 06:04:19.186 INFO (searcherExecutor-15-thread-1-processing-x:uleaf)
[ x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.
2021-01-07 06:04:19.196 INFO (searcherExecutor-15-thread-1-processing-x:uleaf)
[ x:uleaf] o.a.s.c.SolrCore [uleaf] Registered new searcher autowarm time: 1
ms
2021-01-07 06:04:19.198 INFO (qtp1458091526-23) [ x:uleaf]
o.a.s.u.p.LogUpdateProcessorFactory [uleaf] webapp=/solr path=/update
params={waitSearcher=true=true=false=javabin=2}{commit=}
0 228
And the record was duplicated:
The next update generated
2021-01-07 06:10:59.786 INFO (qtp1458091526-17) [ x:uleaf]
o.a.s.u.p.LogUpdateProcessorFactory [uleaf] webapp=/solr path=/update
params={wt=javabin=2}{add=[9901020319M01-X11 (1688207212953993216)]} 0
20
2021-01-07 06:10:59.974 INFO (searcherExecutor-15-thread-1-processing-x:uleaf)
[ x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.
2021-01-07 06:10:59.982 INFO (searcherExecutor-15-thread-1-processing-x:uleaf)
[ x:uleaf] o.a.s.c.SolrCore [uleaf] Registered new searcher autowarm time: 0
ms
2021-01-07 06:10:59.998 INFO (qtp1458091526-26) [ x:uleaf]
o.a.s.u.p.LogUpdateProcessorFactory [uleaf] webapp=/solr path=/update
params={waitSearcher=true=true=false=javabin=2}{commit=}
0 208
Which looks the same as the previous command – so no real difference here.
And then the records looked like
And this shows that the original (7.7.1) item is untouched and only the 8.6.3
item is updated on subsequent updates.
A brand new record being sent to solr generate this dialog
2021-01-07 06:20:10.645 INFO (qtp1458091526-25) [ x:uleaf]
o.a.s.u.p.LogUpdateProcessorFactory [uleaf] webapp=/solr path=/update
params={wt=javabin=2}{add=[9901020319M01-X15 (1688207790576762880),
9901020319M01-DI21 (1688207790587248640)]} 0 15
2021-01-07 06:20:10.798 INFO (searcherExecutor-15-thread-1-processing-x:uleaf)
[ x:uleaf] o.a
RE: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There
As a test I stopped Solr and ran the IndexUpgrader tool on the database to
see if this might fix the issue. It completed OK but unfortunately the issue
still occurs - a new version of the record on solr is created rather than
updating the original record.
It looks to me as if the record created under 7.7.1 is somehow not being
'marked as deleted' in the way that records created under 8.7.0 are. Is
there a way for these records to be marked as deleted when they are updated.
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <mailto:matthew.flower...@unisys.com>
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
<http://www.linkedin.com/company/unisys><http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel>
<http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys>
<http://blogs.unisys.com/>
From: Flowerday, Matthew J
Sent: 07 January 2021 12:25
To: solr-user@lucene.apache.org
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There
I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped
the database and re-indexed (as this would take too long to run on site).
On my local windows machine I have a single solr server 7.7.1 installation
I upgraded in the following manner
* Installed windows solr 8.7.0 on my machine in a different folder
* Copied the core related folder (holding conf, data, lib,
core.properties) from 7.7.1 to the new 8.7.0 folder
* Brought up the solr
* Checked that queries work through the Solr Admin Tool and our
application
This all worked fine until I tried to update a record which had been created
under 7.7.1. Instead of marking the old record as deleted it effectively
created a new copy of the record with the change in and left the old image
as still visible. When I updated the record again it then correctly updated
the new 8.7.0 version without leaving the old image behind. If I created a
new record and then updated it the solr record would be updated correctly.
The issue only seemed to affect the old 7.7.1 created records.
An example of the duplication as follows (the first record is 7.7.1 created
version and the second record is the 8.7.0 version after carrying out an
update):
{
"responseHeader":{
"status":0,
"QTime":4,
"params":{
"q":"id:9901020319M01-N26",
"_":"1610016003669"}},
"response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[
{
"id":"9901020319M01-N26",
"groupId":"9901020319M01",
"urn":"N26",
"specification":"nominal",
"owningGroupId":"9901020319M01",
"description":"N26, Yates, Mike, Alan, Richard, MALE",
"group_t":"9901020319M01",
"nominalUrn_t":"N26",
"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
"dateTimeCreated_dt":"2020-12-30T12:00:53Z",
"title_t":"Captain",
"surname_t":"Yates",
"qualifier_t":"Voyager",
"forename1_t":"Mike",
"forename2_t":"Alan",
"forename3_t":"Richard",
"sex_t":"MALE",
"orderedType_t":"Nominal",
"_version_":1687507566832123904},
{
"id":"9901020319M01-N26",
"groupId":"9901020319M01",
"urn":"N26",
"specification":"nominal",
"owningGroupId":"9901020319M01",
"description":"N26, Yates, Mike, Alan, Richard, MALE",
"group_t":"9901020319M01",
"nominalUrn_t":"N26",
"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
"dateTimeCreated_dt":"2020-12-30T12:00:53Z",
"title_t":"Captain",
"surname_t":"Yates",
"qualifier_t":"Voyager enterprise defiant yorktown xx yy",
"forename1_t":"Mike",
Query over migrating a solr database from 7.7.1 to 8.7.0
Hi There
I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped
the database and re-indexed (as this would take too long to run on site).
On my local windows machine I have a single solr server 7.7.1 installation
I upgraded in the following manner
* Installed windows solr 8.7.0 on my machine in a different folder
* Copied the core related folder (holding conf, data, lib,
core.properties) from 7.7.1 to the new 8.7.0 folder
* Brought up the solr
* Checked that queries work through the Solr Admin Tool and our
application
This all worked fine until I tried to update a record which had been created
under 7.7.1. Instead of marking the old record as deleted it effectively
created a new copy of the record with the change in and left the old image
as still visible. When I updated the record again it then correctly updated
the new 8.7.0 version without leaving the old image behind. If I created a
new record and then updated it the solr record would be updated correctly.
The issue only seemed to affect the old 7.7.1 created records.
An example of the duplication as follows (the first record is 7.7.1 created
version and the second record is the 8.7.0 version after carrying out an
update):
{
"responseHeader":{
"status":0,
"QTime":4,
"params":{
"q":"id:9901020319M01-N26",
"_":"1610016003669"}},
"response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[
{
"id":"9901020319M01-N26",
"groupId":"9901020319M01",
"urn":"N26",
"specification":"nominal",
"owningGroupId":"9901020319M01",
"description":"N26, Yates, Mike, Alan, Richard, MALE",
"group_t":"9901020319M01",
"nominalUrn_t":"N26",
"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
"dateTimeCreated_dt":"2020-12-30T12:00:53Z",
"title_t":"Captain",
"surname_t":"Yates",
"qualifier_t":"Voyager",
"forename1_t":"Mike",
"forename2_t":"Alan",
"forename3_t":"Richard",
"sex_t":"MALE",
"orderedType_t":"Nominal",
"_version_":1687507566832123904},
{
"id":"9901020319M01-N26",
"groupId":"9901020319M01",
"urn":"N26",
"specification":"nominal",
"owningGroupId":"9901020319M01",
"description":"N26, Yates, Mike, Alan, Richard, MALE",
"group_t":"9901020319M01",
"nominalUrn_t":"N26",
"dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
"dateTimeCreated_dt":"2020-12-30T12:00:53Z",
"title_t":"Captain",
"surname_t":"Yates",
"qualifier_t":"Voyager enterprise defiant yorktown xx yy",
"forename1_t":"Mike",
"forename2_t":"Alan",
"forename3_t":"Richard",
"sex_t":"MALE",
"orderedType_t":"Nominal",
"_version_":1688224966566215680}]
}}
I checked the solrconfig.xml file and it does have a uniqueKey set up
id
I was wondering if this behaviour is expected and if there is a way to make
sure that records created under a previous version are updated correctly (so
that the old data is deleted when updated).
Also am I upgrading solr correctly as it could be that the way I have
upgraded it might be causing this issue (I tried hunting through the solr
documentation online but struggled to find window upgrade notes and the
above steps I worked out by trial and error).
Many thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830|
matthew.flower...@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
