Re: optimize boosting parameters

[Radu Gheorghe]

Hi Derek,

Ah, then my reply was completely off :)

I don’t really see a better way. Maybe other than changing termfreq to field, 
if the numeric field has docValues? That may be faster, but I don’t know for 
sure.

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 8 Dec 2020, at 06:17, Derek Poh  wrote:
> 
> Hi Radu
> 
> Apologies for not making myself clear.
> 
> I would like to know if there is a more simple or efficient way to craft the 
> boosting parameters based on the requirements.
> 
> For example, I am using 'if', 'map' and 'termfreq' functions in the bf 
> parameters.
> 
> Is there a more efficient or simple function that can be use instead? Or 
> craft the 'formula' it in a more efficient way?
> 
> On 7/12/2020 10:05 pm, Radu Gheorghe wrote:
>> Hi Derek,
>> 
>> It’s hard to tell whether your boosts can be made better without knowing 
>> your data and what users expect of it. Which is a problem in itself.
>> 
>> I would suggest gathering judgements, like if a user queries for X, what doc 
>> IDs do you expect to get back?
>> 
>> Once you have enough of these judgements, you can experiment with boosts and 
>> see how the query results change. There are measures such as nDCG (
>> https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG
>> ) that can help you measure that per query, and you can average this score 
>> across all your judgements to get an overall measure of how well you’re 
>> doing.
>> 
>> Or even better, you can have something like Quaerite play with boost values 
>> for you:
>> 
>> https://github.com/tballison/quaerite/blob/main/quaerite-examples/README.md#genetic-algorithms-ga-runga
>> 
>> 
>> Best regards,
>> Radu
>> --
>> Sematext Cloud - Full Stack Observability - 
>> https://sematext.com
>> 
>> Solr and Elasticsearch Consulting, Training and Production Support
>> 
>> 
>>> On 7 Dec 2020, at 10:51, Derek Poh 
>>>  wrote:
>>> 
>>> Hi
>>> 
>>> I have added the following boosting requirements to the search query of a 
>>> page. Feedback from monitoring team is that the overall response of the 
>>> page has increased since then.
>>> I am trying to find out if the added boosting parameters (below) could have 
>>> contributed to the increased.
>>> 
>>> The boosting is working as per requirements.
>>> 
>>> May I know if the implemented boosting parameters can be enhanced or 
>>> optimized further?
>>> Hopefully to improve on the response time of the query and the page.
>>> 
>>> Requirements:
>>> 1. If P_SupplierResponseRate is:
>>>a. 3, boost by 0.4
>>>b. 2, boost by 0.2
>>> 
>>> 2. If P_SupplierResponseTime is:
>>>a. 4, boost by 0.4
>>>b. 3, boost by 0.2
>>> 
>>> 3. If P_MWSScore is:
>>>a. between 80-100, boost by 1.6
>>>b. between 60-79, boost by 0.8
>>> 
>>> 4. If P_SupplierRanking is:
>>>a. 3, boost by 0.3
>>>b. 4, boost by 0.6
>>>c. 5, boost by 0.9
>>>b. 6, boost by 1.2
>>> 
>>> Boosting parameters implemented:
>>> bf=map(P_SupplierResponseRate,3,3,0.4,0)
>>> bf=map(P_SupplierResponseRate,2,2,0.2,0)
>>> 
>>> bf=map(P_SupplierResponseTime,4,4,0.4,0)
>>> bf=map(P_SupplierResponseTime,3,3,0.2,0)
>>> 
>>> bf=map(P_MWSScore,80,100,1.6,0)
>>> bf=map(P_MWSScore,60,79,0.8,0)
>>> 
>>> bf=if(termfreq(P_SupplierRanking,3),0.3,if(termfreq(P_SupplierRanking,4),0.6,if(termfreq(P_SupplierRanking,5),0.9,if(termfreq(P_SupplierRanking,6),1.2,0
>>> 
>>> 
>>> I am using Solr 7.7.2
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE 
>>> This e-mail (including any attachments) may contain confidential and/or 
>>> privileged information. If you are not the intended recipient or have 
>>> received this e-mail in error, please inform the sender immediately and 
>>> delete this e-mail (including any attachments) from your computer, and you 
>>> must not use, disclose to anyone else or copy this e-mail (including any 
>>> attachments), whether in whole or in part. 
>>> This e-mail and any reply to it may be monitored for security, legal, 
>>> regulatory compliance and/or other appropriate reasons.
>>> 
>>> 
>> 
> 
> 
> 
> 
> 
> -- 
> CONFIDENTIALITY NOTICE 
> 
> This e-mail (including any attachments) may contain confidential and/or 
> privileged information. If you are not the intended recipient or have 
> received this e-mail in error, please inform the sender immediately and 
> delete this e-mail (including any attachments) from your computer, and you 
> must not use, disclose to anyone else or copy this e-mail (including any 
> attachments), whether in whole or in part. 
> 
> This e-mail and any reply to it may be monitored for security, legal, 
> regulatory compliance and/or other appropriate reasons.
> 
> 

Re: optimize boosting parameters

[Derek Poh]

Hi Radu

Apologies for not making myself clear.

I would like to know if there is a more simple or efficient way to craft 
the boosting parameters based on the requirements.


For example, I am using 'if', 'map' and 'termfreq' functions in the bf 
parameters.


Is there a more efficient or simple function that can be use instead? Or 
craft the 'formula' it in a more efficient way?


On 7/12/2020 10:05 pm, Radu Gheorghe wrote:

Hi Derek,

It’s hard to tell whether your boosts can be made better without knowing your 
data and what users expect of it. Which is a problem in itself.

I would suggest gathering judgements, like if a user queries for X, what doc 
IDs do you expect to get back?

Once you have enough of these judgements, you can experiment with boosts and 
see how the query results change. There are measures such as nDCG 
(https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG) that 
can help you measure that per query, and you can average this score across all 
your judgements to get an overall measure of how well you’re doing.

Or even better, you can have something like Quaerite play with boost values for 
you:
https://github.com/tballison/quaerite/blob/main/quaerite-examples/README.md#genetic-algorithms-ga-runga

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support


On 7 Dec 2020, at 10:51, Derek Poh  wrote:

Hi

I have added the following boosting requirements to the search query of a page. 
Feedback from monitoring team is that the overall response of the page has 
increased since then.
I am trying to find out if the added boosting parameters (below) could have 
contributed to the increased.

The boosting is working as per requirements.

May I know if the implemented boosting parameters can be enhanced or optimized 
further?
Hopefully to improve on the response time of the query and the page.

Requirements:
1. If P_SupplierResponseRate is:
a. 3, boost by 0.4
b. 2, boost by 0.2

2. If P_SupplierResponseTime is:
a. 4, boost by 0.4
b. 3, boost by 0.2

3. If P_MWSScore is:
a. between 80-100, boost by 1.6
b. between 60-79, boost by 0.8

4. If P_SupplierRanking is:
a. 3, boost by 0.3
b. 4, boost by 0.6
c. 5, boost by 0.9
b. 6, boost by 1.2

Boosting parameters implemented:
bf=map(P_SupplierResponseRate,3,3,0.4,0)
bf=map(P_SupplierResponseRate,2,2,0.2,0)

bf=map(P_SupplierResponseTime,4,4,0.4,0)
bf=map(P_SupplierResponseTime,3,3,0.2,0)

bf=map(P_MWSScore,80,100,1.6,0)
bf=map(P_MWSScore,60,79,0.8,0)

bf=if(termfreq(P_SupplierRanking,3),0.3,if(termfreq(P_SupplierRanking,4),0.6,if(termfreq(P_SupplierRanking,5),0.9,if(termfreq(P_SupplierRanking,6),1.2,0


I am using Solr 7.7.2

--
CONFIDENTIALITY NOTICE
This e-mail (including any attachments) may contain confidential and/or 
privileged information. If you are not the intended recipient or have received 
this e-mail in error, please inform the sender immediately and delete this 
e-mail (including any attachments) from your computer, and you must not use, 
disclose to anyone else or copy this e-mail (including any attachments), 
whether in whole or in part.
This e-mail and any reply to it may be monitored for security, legal, 
regulatory compliance and/or other appropriate reasons.






--
CONFIDENTIALITY NOTICE 

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part. 


This e-mail and any reply to it may be monitored for security, legal, 
regulatory compliance and/or other appropriate reasons.

How to get the config set name of Solr core

[Andreas Hubold]

Hi,

is there a way to get the name of the config set for an existing Solr 
core from a stand-alone Solr server (not SolrCloud)?


I need the name of the config set to create another core with the same 
config. The actual use case here is to have a script that creates cores 
of the same config on a Solr slave server for replication. I'm using the 
CREATE action of the CoreAdmin API here. I'd hoped that the CoreAdmin 
STATUS returns the config set, but I cannot find it.


Actually, in previous releases (8.4) the instanceDir returned by 
CoreAdmin STATUS pointed to the config set directory, and I was able to 
derive the config set from that. This seems to have changed now (8.6.3): 
instanceDir seems to point to the directory of the created core now.


Is there a way to get the configSet somehow? Or would it be possible to 
add it to the CoreAdmin STATUS response?


Thank you,
Andreas

Re: is there a way to trigger a notification when a document is deleted in solr

[Walter Underwood]

That wouldn’t help, because that is a feature request to know when the space is
recovered after documents are deleted. 

I’d look at what shows up in the logs when the delete happens. From that info,
you could configure a log follower to notifiy. If your logs go to a log 
database, that
probably supports queries that send notifications.

The original feature request could be satisfied the same way.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 7, 2020, at 6:22 AM, Pushkar Mishra  wrote:
> 
> Hi All
> https://issues.apache.org/jira/browse/SOLR-13609, was this fixed ever ?
> 
> Regards
> 
> On Mon, Dec 7, 2020 at 6:32 PM Pushkar Mishra  wrote:
> 
>> Hi All,
>> 
>> Is there a way to trigger a notification when a document is deleted in
>> solr? Or may be when auto purge gets complete of deleted documents in solr?
>> 
>> Thanks
>> 
>> --
>> Pushkar Kumar Mishra
>> "Reactions are always instinctive whereas responses are always well
>> thought of... So start responding rather than reacting in life"
>> 
>> 
> 
> -- 
> Pushkar Kumar Mishra
> "Reactions are always instinctive whereas responses are always well thought
> of... So start responding rather than reacting in life"

Re: list of all possible values for REQUESTSTATUS

[Ilan Ginzburg]

If you don't want to download the code, you can look at REQUESTSTATUS_OP in
CollectionOperation

(in
CollectionsHandler.java).

Ilan

On Mon, Dec 7, 2020 at 4:57 PM elisabeth benoit 
wrote:

> Hello all,
>
> I'm unloading a core with async param then sending query with request id
>
> http://localhost:8983/solr/admin/cores?action=UNLOAD=expressions=1001http://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001
> and would like to find a piece of doc with all possible values of
> REQUESTSTATUS. Could someone give me a pointer to the doc, I just cant find
> it using a search engine.
>
> I AM NOT looking for
>
> https://lucene.apache.org/solr/guide/8_6/coreadmin-api.html#coreadmin-requeststatus
>
> I would like to have a list of all possible values for STATUS returned
> by solr for
> queryhttp://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001
>
> Is there an available doc or is the only way around to download the
> solr code and search in the code.
>
> Best regards,
> Elisabeth
>

Re: Commits (with openSearcher = true) are too slow in solr 8

[matthew sporleder]

I would stick to soft commits and schedule hard-commits as
spaced-out-as-possible in regular maintenance windows until you can
find the culprit of the timeout.

This way you will have very focused windows for intense monitoring
during the hard-commit runs.


On Mon, Dec 7, 2020 at 9:24 AM raj.yadav  wrote:
>
> Hi Folks,
>
> Do let me know if any more information required to debug this.
>
>
> Regards,
> Raj
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: is there a way to trigger a notification when a document is deleted in solr

[Alexandre Rafalovitch]

Maybe a postCommit listener?
https://lucene.apache.org/solr/guide/8_4/updatehandlers-in-solrconfig.html

Regards,
   Alex.

On Mon, 7 Dec 2020 at 08:03, Pushkar Mishra  wrote:
>
> Hi All,
>
> Is there a way to trigger a notification when a document is deleted in
> solr? Or may be when auto purge gets complete of deleted documents in solr?
>
> Thanks
>
> --
> Pushkar Kumar Mishra
> "Reactions are always instinctive whereas responses are always well thought
> of... So start responding rather than reacting in life"

list of all possible values for REQUESTSTATUS

[elisabeth benoit]

Hello all,

I'm unloading a core with async param then sending query with request id
http://localhost:8983/solr/admin/cores?action=UNLOAD=expressions=1001http://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001
and would like to find a piece of doc with all possible values of
REQUESTSTATUS. Could someone give me a pointer to the doc, I just cant find
it using a search engine.

I AM NOT looking for
https://lucene.apache.org/solr/guide/8_6/coreadmin-api.html#coreadmin-requeststatus

I would like to have a list of all possible values for STATUS returned
by solr for 
queryhttp://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001

Is there an available doc or is the only way around to download the
solr code and search in the code.

Best regards,
Elisabeth

Re: is there a way to trigger a notification when a document is deleted in solr

[Erick Erickson]

No, it’s marked “unresolved”….

> On Dec 7, 2020, at 9:22 AM, Pushkar Mishra  wrote:
> 
> Hi All
> https://issues.apache.org/jira/browse/SOLR-13609, was this fixed ever ?
> 
> Regards
> 
> On Mon, Dec 7, 2020 at 6:32 PM Pushkar Mishra  wrote:
> 
>> Hi All,
>> 
>> Is there a way to trigger a notification when a document is deleted in
>> solr? Or may be when auto purge gets complete of deleted documents in solr?
>> 
>> Thanks
>> 
>> --
>> Pushkar Kumar Mishra
>> "Reactions are always instinctive whereas responses are always well
>> thought of... So start responding rather than reacting in life"
>> 
>> 
> 
> -- 
> Pushkar Kumar Mishra
> "Reactions are always instinctive whereas responses are always well thought
> of... So start responding rather than reacting in life"

Re: Commits (with openSearcher = true) are too slow in solr 8

[raj.yadav]

Hi Folks,

Do let me know if any more information required to debug this.


Regards,
Raj



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: is there a way to trigger a notification when a document is deleted in solr

[Pushkar Mishra]

Hi All
https://issues.apache.org/jira/browse/SOLR-13609, was this fixed ever ?

Regards

On Mon, Dec 7, 2020 at 6:32 PM Pushkar Mishra  wrote:

> Hi All,
>
> Is there a way to trigger a notification when a document is deleted in
> solr? Or may be when auto purge gets complete of deleted documents in solr?
>
> Thanks
>
> --
> Pushkar Kumar Mishra
> "Reactions are always instinctive whereas responses are always well
> thought of... So start responding rather than reacting in life"
>
>

-- 
Pushkar Kumar Mishra
"Reactions are always instinctive whereas responses are always well thought
of... So start responding rather than reacting in life"

Re: doc for REQUESTSTATUS

[Radu Gheorghe]

Hi Elisabeth,

This is the doc for REQUESTSTATUS, apparently only request ID is supported 
indeed: 
https://lucene.apache.org/solr/guide/8_6/coreadmin-api.html#coreadmin-requeststatus

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 7 Dec 2020, at 12:07, elisabeth benoit  wrote:
> 
> Hello all,
> 
> I'm unloading a core with async param then sending query with request id
> 
> http://localhost:8983/solr/admin/cores?action=UNLOAD=expressions=1001
> http://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001
> 
> 
> and would like to find a piece of doc with all possible values of
> REQUESTSTATUS. Could someone give me a pointer to the doc, I just cant find
> it using a search engine.
> 
> Best regards,
> Elisabeth

Re: optimize boosting parameters

[Radu Gheorghe]

Hi Derek,

It’s hard to tell whether your boosts can be made better without knowing your 
data and what users expect of it. Which is a problem in itself.

I would suggest gathering judgements, like if a user queries for X, what doc 
IDs do you expect to get back?

Once you have enough of these judgements, you can experiment with boosts and 
see how the query results change. There are measures such as nDCG 
(https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG) that 
can help you measure that per query, and you can average this score across all 
your judgements to get an overall measure of how well you’re doing.

Or even better, you can have something like Quaerite play with boost values for 
you:
https://github.com/tballison/quaerite/blob/main/quaerite-examples/README.md#genetic-algorithms-ga-runga

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 7 Dec 2020, at 10:51, Derek Poh  wrote:
> 
> Hi
> 
> I have added the following boosting requirements to the search query of a 
> page. Feedback from monitoring team is that the overall response of the page 
> has increased since then.
> I am trying to find out if the added boosting parameters (below) could have 
> contributed to the increased.
> 
> The boosting is working as per requirements.
> 
> May I know if the implemented boosting parameters can be enhanced or 
> optimized further?
> Hopefully to improve on the response time of the query and the page.
> 
> Requirements:
> 1. If P_SupplierResponseRate is:
>a. 3, boost by 0.4
>b. 2, boost by 0.2
> 
> 2. If P_SupplierResponseTime is:
>a. 4, boost by 0.4
>b. 3, boost by 0.2
> 
> 3. If P_MWSScore is:
>a. between 80-100, boost by 1.6
>b. between 60-79, boost by 0.8
> 
> 4. If P_SupplierRanking is:
>a. 3, boost by 0.3
>b. 4, boost by 0.6
>c. 5, boost by 0.9
>b. 6, boost by 1.2
> 
> Boosting parameters implemented:
> bf=map(P_SupplierResponseRate,3,3,0.4,0)
> bf=map(P_SupplierResponseRate,2,2,0.2,0)
> 
> bf=map(P_SupplierResponseTime,4,4,0.4,0)
> bf=map(P_SupplierResponseTime,3,3,0.2,0)
> 
> bf=map(P_MWSScore,80,100,1.6,0)
> bf=map(P_MWSScore,60,79,0.8,0)
> 
> bf=if(termfreq(P_SupplierRanking,3),0.3,if(termfreq(P_SupplierRanking,4),0.6,if(termfreq(P_SupplierRanking,5),0.9,if(termfreq(P_SupplierRanking,6),1.2,0
> 
> 
> I am using Solr 7.7.2
> 
> --
> CONFIDENTIALITY NOTICE 
> This e-mail (including any attachments) may contain confidential and/or 
> privileged information. If you are not the intended recipient or have 
> received this e-mail in error, please inform the sender immediately and 
> delete this e-mail (including any attachments) from your computer, and you 
> must not use, disclose to anyone else or copy this e-mail (including any 
> attachments), whether in whole or in part. 
> This e-mail and any reply to it may be monitored for security, legal, 
> regulatory compliance and/or other appropriate reasons.
> 

Re: What's the most efficient way to check if there are any matches for a query?

[Colvin Cowie]

Thanks for the suggestions. At some point I'll have to actually put it to
the test and see what impact everything has.

Cheers

On Sat, 5 Dec 2020 at 13:31, Erick Erickson  wrote:

> Have you looked at the Term Query Parser (_not_ the TermS Query Parser)
> or Raw Query Parser?
>
> https://lucene.apache.org/solr/guide/8_4/other-parsers.html
>
> NOTE: these perform _no_ analysis, so you have to give them the exact
> term...
>
> These are pretty low level, and if they’re “fast enough” you won’t have to
> do
> any work. You could do some Lucene-level coding I suspect to improve that,
> depends on whether you think those are fast enough…
>
> Best,
> Erick
>
>
> > On Dec 5, 2020, at 5:04 AM, Colvin Cowie 
> wrote:
> >
> > Hello,
> >
> > I was just wondering. If I don't care about the number of matches for a
> > query, let alone what the matches are, just that there is *at least 1*
> > match for a query, what's the most efficient way to execute that query
> (on
> > the /select handler)? (Using Solr 8.7)
> >
> > As a general approach for a query is "rows=0=id asc" the best I can
> > do? Is there a more aggressive short circuit that will stop a searcher as
> > soon as it finds a match?
> >
> > For a specific case where the query is for a single exact term in an
> > indexed field (with or without doc values) is there a different answer?
> >
> > Thanks for any suggestions
>
>

Re: Collection deleted still in zookeeper

[Erick Erickson]

What should happen when you delete a collection and _only_ that
collection references the configset has been discussed several
times, and… whatever is chosen is wrong ;)

1> if we delete the configset, then if you want to delete a collection
to insure that you’re starting all over for whatever reason, your
configset is gone and you need to find it again.

2> If we _don’t_ delete the configset, then you can wind up with
obsolete configsets polluting Zookeeper…

3> If we make a copy of the configset every time we make a collection,
then there can be a bazillion of them in a large installation.

Best,
Erick

> On Dec 7, 2020, at 6:52 AM, Marisol Redondo 
>  wrote:
> 
> Thanks Erick for the answer, you gave me the clue to find the issue.
> 
> The real problem is that when I removed the collection using the solr API
> (http://solrintance:port/solr/admin/collections?action=DELETE=collectionname)
> the config files are not deleted. I don't know if this is the normal
> behavior in every version of solr (I'm using version 6), but I think when
> deleting the collection, the config files for this collection should be
> removed.
> 
> Anyway, I found that the config where still in the UI/cloud/tree/configs
> and they can be removed using the solr zk -r configs/myconfig and this
> solve the issue.
> 
> Thanks
> 
> 
> 
> 
> 
> 
> On Fri, 4 Dec 2020 at 15:46, Erick Erickson  wrote:
> 
>> This almost always a result of one of two things:
>> 
>> 1> you didn’t upload the config to the correct place or the ZK that Solr
>> uses.
>> or
>> 2> you still have a syntax problem in the config.
>> 
>> The solr.log file on the node that’s failing may have a more useful
>> error message about what’s wrong. Also, you can try validating the XML
>> with one of the online tools.
>> 
>> Are you totally and absolutely sure that, for instance, you’re uploading
>> to the correct Zookeeper? You should be able to look at the admin UI
>> screen and see the ZK address. I’ve seen this happen when people
>> inadvertently use the embedded ZK for one operation but not for the
>> other. Of have the ZK_HOST environment variable pointing to some
>> ZK ensemble that’s used when you start Solr but not when you upload
>> files. Or…
>> 
>> Use the admin UI>>cloud>>tree>>configs>>your_config_name
>> to see if the solrconfig has the correct changes. I’ll often add some
>> bogus comment in the early part of the file that I can use to make
>> sure I’ve uploaded the correct file to the correct place.
>> 
>> I use the "bin/solr zk upconfig” command to move files back and forth
>> FWIW, that
>> avoids, say putting the individual file a in the wrong directory...
>> 
>> Best,
>> Erick
>> 
>>> On Dec 4, 2020, at 9:18 AM, Marisol Redondo <
>> marisol.redondo.gar...@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> When trying to modify the config.xml file for a collection I made a
>> mistake
>>> and the config was wrong. So I removed the collection to create it again
>>> from a backend.
>>> But, although I'm sure I'm using a correct config.xml, solr is still
>>> complaining about the error in the older solrconfig.xml
>>> 
>>> I have tried to removed the collection more than once, I have stopped
>> solr
>>> and zookeeper and still having the same error. It's like zookeeper is
>> still
>>> storing the older solrconfig.xml and don't upload the configuration file
>>> from the new collection.
>>> 
>>> I have tried to
>>> - upload the files
>>> - remove the collection and create it again, but empty
>>> - restore the collection from the backup
>>> And I get always the same error:
>>>  collection_name_shard1_replica1:
>>> 
>> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
>>> Could not load conf for core collection_name_shard1_replica1: Error
>> loading
>>> solr config from solrconfig.xml
>>> 
>>> Thanks for your help
>> 
>> 

is there a way to trigger a notification when a document is deleted in solr

[Pushkar Mishra]

Hi All,

Is there a way to trigger a notification when a document is deleted in
solr? Or may be when auto purge gets complete of deleted documents in solr?

Thanks

-- 
Pushkar Kumar Mishra
"Reactions are always instinctive whereas responses are always well thought
of... So start responding rather than reacting in life"

Re: Collection deleted still in zookeeper

[Marisol Redondo]

Thanks Erick for the answer, you gave me the clue to find the issue.

The real problem is that when I removed the collection using the solr API
(http://solrintance:port/solr/admin/collections?action=DELETE=collectionname)
the config files are not deleted. I don't know if this is the normal
behavior in every version of solr (I'm using version 6), but I think when
deleting the collection, the config files for this collection should be
removed.

Anyway, I found that the config where still in the UI/cloud/tree/configs
and they can be removed using the solr zk -r configs/myconfig and this
solve the issue.

Thanks






On Fri, 4 Dec 2020 at 15:46, Erick Erickson  wrote:

> This almost always a result of one of two things:
>
> 1> you didn’t upload the config to the correct place or the ZK that Solr
> uses.
> or
> 2> you still have a syntax problem in the config.
>
> The solr.log file on the node that’s failing may have a more useful
> error message about what’s wrong. Also, you can try validating the XML
> with one of the online tools.
>
> Are you totally and absolutely sure that, for instance, you’re uploading
> to the correct Zookeeper? You should be able to look at the admin UI
> screen and see the ZK address. I’ve seen this happen when people
> inadvertently use the embedded ZK for one operation but not for the
> other. Of have the ZK_HOST environment variable pointing to some
> ZK ensemble that’s used when you start Solr but not when you upload
> files. Or…
>
> Use the admin UI>>cloud>>tree>>configs>>your_config_name
> to see if the solrconfig has the correct changes. I’ll often add some
> bogus comment in the early part of the file that I can use to make
> sure I’ve uploaded the correct file to the correct place.
>
> I use the "bin/solr zk upconfig” command to move files back and forth
> FWIW, that
> avoids, say putting the individual file a in the wrong directory...
>
> Best,
> Erick
>
> > On Dec 4, 2020, at 9:18 AM, Marisol Redondo <
> marisol.redondo.gar...@gmail.com> wrote:
> >
> > Hi,
> >
> > When trying to modify the config.xml file for a collection I made a
> mistake
> > and the config was wrong. So I removed the collection to create it again
> > from a backend.
> > But, although I'm sure I'm using a correct config.xml, solr is still
> > complaining about the error in the older solrconfig.xml
> >
> > I have tried to removed the collection more than once, I have stopped
> solr
> > and zookeeper and still having the same error. It's like zookeeper is
> still
> > storing the older solrconfig.xml and don't upload the configuration file
> > from the new collection.
> >
> > I have tried to
> > - upload the files
> > - remove the collection and create it again, but empty
> > - restore the collection from the backup
> > And I get always the same error:
> >   collection_name_shard1_replica1:
> >
> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> > Could not load conf for core collection_name_shard1_replica1: Error
> loading
> > solr config from solrconfig.xml
> >
> > Thanks for your help
>
>

Migrating from solr 7.7 to solr 8.6 issues

[Abhishek Mishra]

We are trying to migrate from solr 7.7 to solr 8.6 on Kubernetes. We are
using zookeeper-3.4.13. While adding a replica to the cluster, it returns
500 status code. While in the background it is added sometimes successfully
while sometime it is in the inactive node. We are using http2 without SSL.

Error:

>  {

  "responseHeader":{
"status":500,
"QTime":307},
  "failure":{

"solr-pklatest-statefulset-pull-0.solr-pklatest-statefulset-headless.relevance:8983_solr":"org.apache.solr.client.solrj.SolrServerException:IOException
occured when talking to server at: null"},
  "Operation addreplica caused
exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
ADDREPLICA failed to create replica",
  "exception":{
"msg":"ADDREPLICA failed to create replica",
"rspCode":500},
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","org.apache.solr.common.SolrException"],
"msg":"ADDREPLICA failed to create replica",
"trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to
create replica\n\tat
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:65)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:286)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:257)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:854)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:818)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)\n\tat
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat

doc for REQUESTSTATUS

[elisabeth benoit]

Hello all,

I'm unloading a core with async param then sending query with request id

http://localhost:8983/solr/admin/cores?action=UNLOAD=expressions=1001
http://localhost:8983/solr/admin/cores?action=REQUESTSTATUS=1001


and would like to find a piece of doc with all possible values of
REQUESTSTATUS. Could someone give me a pointer to the doc, I just cant find
it using a search engine.

Best regards,
Elisabeth

Inconsistent recovery status of replicas

[Abhishek Mishra]

Hello guys
I am using Solr cloud 7.7 on Kubernetes. During the adding of replica
sometimes we see inconsistency after successful addition nodes go to
recovery status sometimes it takes 2-3 minute to recover while sometimes it
takes more than an hour. We are getting this error.
We have 4 shards each shard has around 7GB of data. After seeing the system
metrics we see bandwidth exchanges are high between the leader and the new
replica node. Do we have any way to rate-limit the bandwidth exchange like
we had some configuration for it in master-slave? maxMbpersec something
like that?

Error

> 2020-12-01 13:40:34.983 ERROR 
> (recoveryExecutor-4-thread-1-processing-n:solr-olxid-statefulset-pull-9.solr-olxid-statefulset-headless.relevance:8983_solr
>  x:olxid-20200531_d6e431ec_shard2_replica_p3955 c:olxid-20200531_d6e431ec 
> s:shard2 r:core_node3956) [c:olxid-20200531_d6e431ec s:shard2 r:core_node3956 
> x:olxid-20200531_d6e431ec_shard2_replica_p3955] o.a.s.c.RecoveryStrategy 
> Error while trying to 
> recover:org.apache.solr.client.solrj.SolrServerException: Timeout occured 
> while waiting response from server at: 
> http://solr-olxid-statefulset-tlog-7.solr-olxid-statefulset-headless.relevance:8983/solr/olxid-20200531_d6e431ec_shard2_replica_t139
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>   at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>   at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:287)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:215)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)
>   at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)
>   at 
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>   at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.net.SocketTimeoutException: Read timed out
>   at java.base/java.net.SocketInputStream.socketRead0(Native Method)
>   at 
> java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
>   at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
>   at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
>   at 
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
>   at 
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
>   at 
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
>   at 
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
>   at 
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
>   at 
> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120)
>   at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
>   at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>   at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>   at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>   at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

optimize boosting parameters

[Derek Poh]

Hi

I have added the following boosting requirements to the search query of 
a page. Feedback from monitoring team is that the overall response of 
the page has increased since then.
I am trying to find out if the added boosting parameters (below) could 
have contributed to the increased.


The boosting is working as per requirements.

May I know if the implemented boosting parameters can be enhanced or 
optimized further?

Hopefully to improve on the response time of the query and the page.

Requirements:
1. If P_SupplierResponseRate is:
   a. 3, boost by 0.4
   b. 2, boost by 0.2

2. If P_SupplierResponseTime is:
   a. 4, boost by 0.4
   b. 3, boost by 0.2

3. If P_MWSScore is:
   a. between 80-100, boost by 1.6
   b. between 60-79, boost by 0.8

4. If P_SupplierRanking is:
   a. 3, boost by 0.3
   b. 4, boost by 0.6
   c. 5, boost by 0.9
   b. 6, boost by 1.2

Boosting parameters implemented:
bf=map(P_SupplierResponseRate,3,3,0.4,0)
bf=map(P_SupplierResponseRate,2,2,0.2,0)

bf=map(P_SupplierResponseTime,4,4,0.4,0)
bf=map(P_SupplierResponseTime,3,3,0.2,0)

bf=map(P_MWSScore,80,100,1.6,0)
bf=map(P_MWSScore,60,79,0.8,0)

bf=if(termfreq(P_SupplierRanking,3),0.3,if(termfreq(P_SupplierRanking,4),0.6,if(termfreq(P_SupplierRanking,5),0.9,if(termfreq(P_SupplierRanking,6),1.2,0


I am using Solr 7.7.2

--
CONFIDENTIALITY NOTICE 

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part. 


This e-mail and any reply to it may be monitored for security, legal, 
regulatory compliance and/or other appropriate reasons.