RE: [SPAM] Re: query parsed in different ways in two identical solr instances
Yes I identical because the configuration (solrconfig.xml etc) is identical,
just some fields changed.
Sorry I was not so precise in the description of the environment.
Nice to know it's already fixed.
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
As for the European General Data Protection Regulation 2016/679 on the
protection of natural persons with regard to the processing of personal data,
we inform you that all the data we possess are object of treatment in the
respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may
ask for their correction, cancellation or you may oppose to their use by
written request sent by recorded delivery to The Microsoft Research –
University of Trento Centre for Computational and Systems Biology Scarl, Piazza
Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to
From: Alexandre Rafalovitch [arafa...@gmail.com]
Sent: 10 June 2019 15:32
To: solr-user
Subject: Re: [SPAM] Re: query parsed in different ways in two identical solr
instances
Ok, great.
We now moved from "identical setup breaks things in a bugfix version"
to "strange behavior when field does not exist". The "identical" part
was actually throwing us off the trail.
And all this leads us to
https://issues.apache.org/jira/browse/SOLR-5163 , fixed in 8.0.
Hope it helps,
Alex.
On Mon, 10 Jun 2019 at 09:19, Danilo Tomasoni wrote:
>
> Hello I was able to reproduce this behaviour in an isolated environment,
> and performed some differential analysis between the two versions (that has
> different schemas, diff of schemas attached)
>
> With the schema of solr1, the query is parsed as +(+() +())
> while with the schema of solr-test, the same query is parsed as +(()
> ())
>
> The query is
>
> "q":"(f1:PUBMEDPMID12159614 AND (_query_:\"{!edismax
> qf='medline_chemical_terms medline_mesh_terms' q.op=OR mm=1 v=$subquery1}\"))"
>
> in solr1 and also in solr test f1 equals
> "f.f1.qf":"id pmid pmc source_id other_id doi manuscript_id publication_id
> secondary_ids"}}
>
> And then I suddenly remembered that the field secondary_ids was renamed to
> external_data in solr-test (before the bulk import).
>
> So I changed f1 definition removing secondary_ids and adding external_data..
> and now the behaviour is the same!
>
> How is that possible? why the schema (and in this case a non-existing field)
> can influence in such a profound way the behaviour of the query parser?
>
> I think that this is a subtle bug and an error should be raised instead of
> performing an unexpected query.
>
> Danilo Tomasoni
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal data,
> we inform you that all the data we possess are object of treatment in the
> respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you
> may ask for their correction, cancellation or you may oppose to their use by
> written request sent by recorded delivery to The Microsoft Research –
> University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
>
> From: Alexandre Rafalovitch [arafa...@gmail.com]
> Sent: 10 June 2019 12:49
> To: solr-user
> Subject: [SPAM] Re: query parsed in different ways in two identical solr
> instances
>
> Were you able to simplify it to the simplest use case showing the issue? Or
> reproduce it on the stock Solr with stock example? Because otherwise, we
> would be just as stuck in a Jira as now. It is the same people helping
>
> For example, is the _query_ part significant?
>
> Also, did you try running both queries with echoParams=all just to
> eliminate stray differences? I know you looked at the debug line, but
> perhaps this is worth a check too.
>
> Regards,
> Alex
>
>
>
> On Mon, Jun 10, 2019, 5:46 AM Danilo Tomasoni, wrote:
>
> > Hello all,
> > maybe I should consider this as a bug and open an issue?
> &
Re: [SPAM] Re: query parsed in different ways in two identical solr instances
Ok, great.
We now moved from "identical setup breaks things in a bugfix version"
to "strange behavior when field does not exist". The "identical" part
was actually throwing us off the trail.
And all this leads us to
https://issues.apache.org/jira/browse/SOLR-5163 , fixed in 8.0.
Hope it helps,
Alex.
On Mon, 10 Jun 2019 at 09:19, Danilo Tomasoni wrote:
>
> Hello I was able to reproduce this behaviour in an isolated environment,
> and performed some differential analysis between the two versions (that has
> different schemas, diff of schemas attached)
>
> With the schema of solr1, the query is parsed as +(+() +())
> while with the schema of solr-test, the same query is parsed as +(()
> ())
>
> The query is
>
> "q":"(f1:PUBMEDPMID12159614 AND (_query_:\"{!edismax
> qf='medline_chemical_terms medline_mesh_terms' q.op=OR mm=1 v=$subquery1}\"))"
>
> in solr1 and also in solr test f1 equals
> "f.f1.qf":"id pmid pmc source_id other_id doi manuscript_id publication_id
> secondary_ids"}}
>
> And then I suddenly remembered that the field secondary_ids was renamed to
> external_data in solr-test (before the bulk import).
>
> So I changed f1 definition removing secondary_ids and adding external_data..
> and now the behaviour is the same!
>
> How is that possible? why the schema (and in this case a non-existing field)
> can influence in such a profound way the behaviour of the query parser?
>
> I think that this is a subtle bug and an error should be raised instead of
> performing an unexpected query.
>
> Danilo Tomasoni
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal data,
> we inform you that all the data we possess are object of treatment in the
> respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you
> may ask for their correction, cancellation or you may oppose to their use by
> written request sent by recorded delivery to The Microsoft Research –
> University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
>
> From: Alexandre Rafalovitch [arafa...@gmail.com]
> Sent: 10 June 2019 12:49
> To: solr-user
> Subject: [SPAM] Re: query parsed in different ways in two identical solr
> instances
>
> Were you able to simplify it to the simplest use case showing the issue? Or
> reproduce it on the stock Solr with stock example? Because otherwise, we
> would be just as stuck in a Jira as now. It is the same people helping
>
> For example, is the _query_ part significant?
>
> Also, did you try running both queries with echoParams=all just to
> eliminate stray differences? I know you looked at the debug line, but
> perhaps this is worth a check too.
>
> Regards,
> Alex
>
>
>
> On Mon, Jun 10, 2019, 5:46 AM Danilo Tomasoni, wrote:
>
> > Hello all,
> > maybe I should consider this as a bug and open an issue?
> >
> > Danilo Tomasoni
> >
> > Fondazione The Microsoft Research - University of Trento Centre for
> > Computational and Systems Biology (COSBI)
> > Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> > tomas...@cosbi.eu
> > http://www.cosbi.eu
> >
> > As for the European General Data Protection Regulation 2016/679 on the
> > protection of natural persons with regard to the processing of personal
> > data, we inform you that all the data we possess are object of treatment in
> > the respect of the normative provided for by the cited GDPR.
> > It is your right to be informed on which of your data are used and how;
> > you may ask for their correction, cancellation or you may oppose to their
> > use by written request sent by recorded delivery to The Microsoft Research
> > – University of Trento Centre for Computational and Systems Biology Scarl,
> > Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> > P Please don't print this e-mail unless you really need to
> >
> >
> > From: Danilo Tomasoni
> > Sent: 07 June 2019 11:47
> > To: solr-user@lucene.apache.org
> > Subject: RE: query parsed in different ways in two identical solr instance
RE: [SPAM] Re: query parsed in different ways in two identical solr instances
Hello I was able to reproduce this behaviour in an isolated environment,
and performed some differential analysis between the two versions (that has
different schemas, diff of schemas attached)
With the schema of solr1, the query is parsed as +(+() +())
while with the schema of solr-test, the same query is parsed as +(() ())
The query is
"q":"(f1:PUBMEDPMID12159614 AND (_query_:\"{!edismax qf='medline_chemical_terms
medline_mesh_terms' q.op=OR mm=1 v=$subquery1}\"))"
in solr1 and also in solr test f1 equals
"f.f1.qf":"id pmid pmc source_id other_id doi manuscript_id publication_id
secondary_ids"}}
And then I suddenly remembered that the field secondary_ids was renamed to
external_data in solr-test (before the bulk import).
So I changed f1 definition removing secondary_ids and adding external_data..
and now the behaviour is the same!
How is that possible? why the schema (and in this case a non-existing field)
can influence in such a profound way the behaviour of the query parser?
I think that this is a subtle bug and an error should be raised instead of
performing an unexpected query.
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
As for the European General Data Protection Regulation 2016/679 on the
protection of natural persons with regard to the processing of personal data,
we inform you that all the data we possess are object of treatment in the
respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may
ask for their correction, cancellation or you may oppose to their use by
written request sent by recorded delivery to The Microsoft Research –
University of Trento Centre for Computational and Systems Biology Scarl, Piazza
Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to
From: Alexandre Rafalovitch [arafa...@gmail.com]
Sent: 10 June 2019 12:49
To: solr-user
Subject: [SPAM] Re: query parsed in different ways in two identical solr
instances
Were you able to simplify it to the simplest use case showing the issue? Or
reproduce it on the stock Solr with stock example? Because otherwise, we
would be just as stuck in a Jira as now. It is the same people helping
For example, is the _query_ part significant?
Also, did you try running both queries with echoParams=all just to
eliminate stray differences? I know you looked at the debug line, but
perhaps this is worth a check too.
Regards,
Alex
On Mon, Jun 10, 2019, 5:46 AM Danilo Tomasoni, wrote:
> Hello all,
> maybe I should consider this as a bug and open an issue?
>
> Danilo Tomasoni
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal
> data, we inform you that all the data we possess are object of treatment in
> the respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how;
> you may ask for their correction, cancellation or you may oppose to their
> use by written request sent by recorded delivery to The Microsoft Research
> – University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
> ________
> From: Danilo Tomasoni
> Sent: 07 June 2019 11:47
> To: solr-user@lucene.apache.org
> Subject: RE: query parsed in different ways in two identical solr instances
>
> any thoughts on that difference in the solr parsing? is it correct that
> the first looks like an AND while the second looks like and OR?
> Thank you
>
> Danilo Tomasoni
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal
> data, we inform you that all the data we possess are object of treatment in
> the respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how;
> you may ask for their correction, cancellation or you may oppo
Re: query parsed in different ways in two identical solr instances
Were you able to simplify it to the simplest use case showing the issue? Or reproduce it on the stock Solr with stock example? Because otherwise, we would be just as stuck in a Jira as now. It is the same people helping For example, is the _query_ part significant? Also, did you try running both queries with echoParams=all just to eliminate stray differences? I know you looked at the debug line, but perhaps this is worth a check too. Regards, Alex On Mon, Jun 10, 2019, 5:46 AM Danilo Tomasoni, wrote: > Hello all, > maybe I should consider this as a bug and open an issue? > > Danilo Tomasoni > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu > http://www.cosbi.eu > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by written request sent by recorded delivery to The Microsoft Research > – University of Trento Centre for Computational and Systems Biology Scarl, > Piazza Manifattura 1, 38068 Rovereto (TN), Italy. > P Please don't print this e-mail unless you really need to > > > From: Danilo Tomasoni > Sent: 07 June 2019 11:47 > To: solr-user@lucene.apache.org > Subject: RE: query parsed in different ways in two identical solr instances > > any thoughts on that difference in the solr parsing? is it correct that > the first looks like an AND while the second looks like and OR? > Thank you > > Danilo Tomasoni > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu > http://www.cosbi.eu > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by written request sent by recorded delivery to The Microsoft Research > – University of Trento Centre for Computational and Systems Biology Scarl, > Piazza Manifattura 1, 38068 Rovereto (TN), Italy. > P Please don't print this e-mail unless you really need to > > ________ > From: Danilo Tomasoni [tomas...@cosbi.eu] > Sent: 06 June 2019 16:21 > To: solr-user@lucene.apache.org > Subject: RE: query parsed in different ways in two identical solr instances > > The two collections are not identical, many overlapping documents but with > some different field names (test has also extra fields that 1 didn't have). > Actually we have 42.000.000 docs in solr1, and 40.000.000 in solr-test, > but I think this shouldn'd be relevant because the query is basically like > > id=x AND mesh=list of phrase queries > > where the second part of the and is handled through a nested query > (_query_ magic keyword). > > I expect that a query like this one would return 1 documents (x) or 0 > documents. > > The thing that puzzles me is that on solr1 the engine is returning 1 > document (x) > while on test the engine is returning 68.000 documents.. > If you look at my first e-mail you will notice that in the correct engine > the parsed query is like > > +(+(...) +(...)) > > That is correct for an AND > > while in the test engine the query is parsed like > > +((...) (...)) > > which is more like an OR... > > > Danilo Tomasoni > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu > http://www.cosbi.eu > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by wri
RE: query parsed in different ways in two identical solr instances
Hello all, maybe I should consider this as a bug and open an issue? Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu http://www.cosbi.eu As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to From: Danilo Tomasoni Sent: 07 June 2019 11:47 To: solr-user@lucene.apache.org Subject: RE: query parsed in different ways in two identical solr instances any thoughts on that difference in the solr parsing? is it correct that the first looks like an AND while the second looks like and OR? Thank you Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu http://www.cosbi.eu As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to From: Danilo Tomasoni [tomas...@cosbi.eu] Sent: 06 June 2019 16:21 To: solr-user@lucene.apache.org Subject: RE: query parsed in different ways in two identical solr instances The two collections are not identical, many overlapping documents but with some different field names (test has also extra fields that 1 didn't have). Actually we have 42.000.000 docs in solr1, and 40.000.000 in solr-test, but I think this shouldn'd be relevant because the query is basically like id=x AND mesh=list of phrase queries where the second part of the and is handled through a nested query (_query_ magic keyword). I expect that a query like this one would return 1 documents (x) or 0 documents. The thing that puzzles me is that on solr1 the engine is returning 1 document (x) while on test the engine is returning 68.000 documents.. If you look at my first e-mail you will notice that in the correct engine the parsed query is like +(+(...) +(...)) That is correct for an AND while in the test engine the query is parsed like +((...) (...)) which is more like an OR... Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu http://www.cosbi.eu As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to From: Alexandre Rafalovitch [arafa...@gmail.com] Sent: 06 June 2019 15:53 To: solr-user Subject: Re: query parsed in different ways in two identical solr instances Those two queries look same after sorting the parameters, yet the results are clearly different. That means the difference is deeper. 1) Have you checked that both collections have the same amount of documents (e.g. mismatched final commit). Does basic "query=*:*" return the same counts in the same initial order? 2) Are you absolutely sure you are comparing 7.3.0 with 7.3.1? There was SOLR-11501 that may be relevant, but it was fixed in 7.2: https://issues.apache.org/jira/browse/SOLR-11501 Regards, Alex
RE: query parsed in different ways in two identical solr instances
any thoughts on that difference in the solr parsing? is it correct that the first looks like an AND while the second looks like and OR? Thank you Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu http://www.cosbi.eu As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to From: Danilo Tomasoni [tomas...@cosbi.eu] Sent: 06 June 2019 16:21 To: solr-user@lucene.apache.org Subject: RE: query parsed in different ways in two identical solr instances The two collections are not identical, many overlapping documents but with some different field names (test has also extra fields that 1 didn't have). Actually we have 42.000.000 docs in solr1, and 40.000.000 in solr-test, but I think this shouldn'd be relevant because the query is basically like id=x AND mesh=list of phrase queries where the second part of the and is handled through a nested query (_query_ magic keyword). I expect that a query like this one would return 1 documents (x) or 0 documents. The thing that puzzles me is that on solr1 the engine is returning 1 document (x) while on test the engine is returning 68.000 documents.. If you look at my first e-mail you will notice that in the correct engine the parsed query is like +(+(...) +(...)) That is correct for an AND while in the test engine the query is parsed like +((...) (...)) which is more like an OR... Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1, 38068 Rovereto (TN), Italy tomas...@cosbi.eu http://www.cosbi.eu As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR. It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy. P Please don't print this e-mail unless you really need to From: Alexandre Rafalovitch [arafa...@gmail.com] Sent: 06 June 2019 15:53 To: solr-user Subject: Re: query parsed in different ways in two identical solr instances Those two queries look same after sorting the parameters, yet the results are clearly different. That means the difference is deeper. 1) Have you checked that both collections have the same amount of documents (e.g. mismatched final commit). Does basic "query=*:*" return the same counts in the same initial order? 2) Are you absolutely sure you are comparing 7.3.0 with 7.3.1? There was SOLR-11501 that may be relevant, but it was fixed in 7.2: https://issues.apache.org/jira/browse/SOLR-11501 Regards, Alex. Are you absolutely sure that your instances are 7.3.0 and 7.3.1? On Thu, 6 Jun 2019 at 09:26, Danilo Tomasoni wrote: > > Hello, and thank you for your answer. > Attached you will find the two logs for the working solr1 server, and the > non-working solr-test server. > > > Danilo Tomasoni > > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu > http://www.cosbi.eu > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal data, > we inform you that all the data we possess are object of treatment in the > respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; you > may ask for their correction, cancellation or you may oppose to their use by > written request sent by recorded delivery to The Microsoft Research – > University of Trento Centre for Computational and Systems Biology Scarl, > Pi
RE: query parsed in different ways in two identical solr instances
ah yes, I'm sure we are using solr 7.3.1 as test (non working) and solr 7.3.0
as 1 (working)
7.3.0 98a6b3d642928b1ac9076c6c5a369472581f7633 - woody - 2018-03-28 14:37:45
vs
7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09
09:30:57
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
As for the European General Data Protection Regulation 2016/679 on the
protection of natural persons with regard to the processing of personal data,
we inform you that all the data we possess are object of treatment in the
respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may
ask for their correction, cancellation or you may oppose to their use by
written request sent by recorded delivery to The Microsoft Research –
University of Trento Centre for Computational and Systems Biology Scarl, Piazza
Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to
From: Alexandre Rafalovitch [arafa...@gmail.com]
Sent: 06 June 2019 15:53
To: solr-user
Subject: Re: query parsed in different ways in two identical solr instances
Those two queries look same after sorting the parameters, yet the
results are clearly different. That means the difference is deeper.
1) Have you checked that both collections have the same amount of
documents (e.g. mismatched final commit). Does basic "query=*:*"
return the same counts in the same initial order?
2) Are you absolutely sure you are comparing 7.3.0 with 7.3.1? There
was SOLR-11501 that may be relevant, but it was fixed in 7.2:
https://issues.apache.org/jira/browse/SOLR-11501
Regards,
Alex.
Are you absolutely sure that your instances are 7.3.0 and 7.3.1?
On Thu, 6 Jun 2019 at 09:26, Danilo Tomasoni wrote:
>
> Hello, and thank you for your answer.
> Attached you will find the two logs for the working solr1 server, and the
> non-working solr-test server.
>
>
> Danilo Tomasoni
>
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal data,
> we inform you that all the data we possess are object of treatment in the
> respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you
> may ask for their correction, cancellation or you may oppose to their use by
> written request sent by recorded delivery to The Microsoft Research –
> University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
>
> From: Shawn Heisey [apa...@elyograg.org]
> Sent: 05 June 2019 17:52
> To: solr-user@lucene.apache.org
> Subject: Re: query parsed in different ways in two identical solr instances
>
> On 6/5/2019 8:41 AM, Danilo Tomasoni wrote:
> > Hello,
> > I have two solr instances with exactly the same configuration.
> > The only difference that i know is that the first (the working one, is solr
> > 7.3.0,
> > while the one that's not working is solr 7.3.1)
> >
> > If I execute the same query (with debugQuery=on) it gets parsed in
> > different ways on the two systems and I don't understand why.
>
> Look in solr.log. The full query, including parameters that are used
> but not on the URL, will be shown there. Provide that whole line from
> both versions.
>
> An example of the kind of line you need to find, with a very simple
> query, is below:
>
> 2019-06-05 15:50:23.691 INFO (qtp1264413185-43) [ x:foo]
> o.a.s.c.S.Request [foo] webapp=/solr path=/select
> params={q=*:*&_=1559749821933} hits=0 status=0 QTime=38
>
> If your index has multiple shards, there can be multiple lines. In that
> situation, we need the last one, which should be the main query itself
> rather than the subqueries.
>
> Thanks,
> Shawn
RE: query parsed in different ways in two identical solr instances
The two collections are not identical, many overlapping documents but with some
different field names (test has also extra fields that 1 didn't have).
Actually we have 42.000.000 docs in solr1, and 40.000.000 in solr-test, but I
think this shouldn'd be relevant because the query is basically like
id=x AND mesh=list of phrase queries
where the second part of the and is handled through a nested query (_query_
magic keyword).
I expect that a query like this one would return 1 documents (x) or 0 documents.
The thing that puzzles me is that on solr1 the engine is returning 1 document
(x)
while on test the engine is returning 68.000 documents..
If you look at my first e-mail you will notice that in the correct engine the
parsed query is like
+(+(...) +(...))
That is correct for an AND
while in the test engine the query is parsed like
+((...) (...))
which is more like an OR...
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
As for the European General Data Protection Regulation 2016/679 on the
protection of natural persons with regard to the processing of personal data,
we inform you that all the data we possess are object of treatment in the
respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may
ask for their correction, cancellation or you may oppose to their use by
written request sent by recorded delivery to The Microsoft Research –
University of Trento Centre for Computational and Systems Biology Scarl, Piazza
Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to
From: Alexandre Rafalovitch [arafa...@gmail.com]
Sent: 06 June 2019 15:53
To: solr-user
Subject: Re: query parsed in different ways in two identical solr instances
Those two queries look same after sorting the parameters, yet the
results are clearly different. That means the difference is deeper.
1) Have you checked that both collections have the same amount of
documents (e.g. mismatched final commit). Does basic "query=*:*"
return the same counts in the same initial order?
2) Are you absolutely sure you are comparing 7.3.0 with 7.3.1? There
was SOLR-11501 that may be relevant, but it was fixed in 7.2:
https://issues.apache.org/jira/browse/SOLR-11501
Regards,
Alex.
Are you absolutely sure that your instances are 7.3.0 and 7.3.1?
On Thu, 6 Jun 2019 at 09:26, Danilo Tomasoni wrote:
>
> Hello, and thank you for your answer.
> Attached you will find the two logs for the working solr1 server, and the
> non-working solr-test server.
>
>
> Danilo Tomasoni
>
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal data,
> we inform you that all the data we possess are object of treatment in the
> respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you
> may ask for their correction, cancellation or you may oppose to their use by
> written request sent by recorded delivery to The Microsoft Research –
> University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
>
> From: Shawn Heisey [apa...@elyograg.org]
> Sent: 05 June 2019 17:52
> To: solr-user@lucene.apache.org
> Subject: Re: query parsed in different ways in two identical solr instances
>
> On 6/5/2019 8:41 AM, Danilo Tomasoni wrote:
> > Hello,
> > I have two solr instances with exactly the same configuration.
> > The only difference that i know is that the first (the working one, is solr
> > 7.3.0,
> > while the one that's not working is solr 7.3.1)
> >
> > If I execute the same query (with debugQuery=on) it gets parsed in
> > different ways on the two systems and I don't understand why.
>
> Look in solr.log. The full query, including parameters that are used
> but not on the URL, will be shown there. Provide that whole line from
> both versions.
>
> An example of the kind of line you need to find, with a very simple
> query, is below:
>
> 2019-06-05 15:50:23.691 INFO (qtp1264413185-43) [ x:foo]
> o.a.s.c.S.Request [foo] webapp=/solr path=/select
> params={q=*:*&_=1559749821933} hits=0 st
Re: query parsed in different ways in two identical solr instances
Those two queries look same after sorting the parameters, yet the
results are clearly different. That means the difference is deeper.
1) Have you checked that both collections have the same amount of
documents (e.g. mismatched final commit). Does basic "query=*:*"
return the same counts in the same initial order?
2) Are you absolutely sure you are comparing 7.3.0 with 7.3.1? There
was SOLR-11501 that may be relevant, but it was fixed in 7.2:
https://issues.apache.org/jira/browse/SOLR-11501
Regards,
Alex.
Are you absolutely sure that your instances are 7.3.0 and 7.3.1?
On Thu, 6 Jun 2019 at 09:26, Danilo Tomasoni wrote:
>
> Hello, and thank you for your answer.
> Attached you will find the two logs for the working solr1 server, and the
> non-working solr-test server.
>
>
> Danilo Tomasoni
>
>
> Fondazione The Microsoft Research - University of Trento Centre for
> Computational and Systems Biology (COSBI)
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy
> tomas...@cosbi.eu
> http://www.cosbi.eu
>
> As for the European General Data Protection Regulation 2016/679 on the
> protection of natural persons with regard to the processing of personal data,
> we inform you that all the data we possess are object of treatment in the
> respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you
> may ask for their correction, cancellation or you may oppose to their use by
> written request sent by recorded delivery to The Microsoft Research –
> University of Trento Centre for Computational and Systems Biology Scarl,
> Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>
>
> From: Shawn Heisey [apa...@elyograg.org]
> Sent: 05 June 2019 17:52
> To: solr-user@lucene.apache.org
> Subject: Re: query parsed in different ways in two identical solr instances
>
> On 6/5/2019 8:41 AM, Danilo Tomasoni wrote:
> > Hello,
> > I have two solr instances with exactly the same configuration.
> > The only difference that i know is that the first (the working one, is solr
> > 7.3.0,
> > while the one that's not working is solr 7.3.1)
> >
> > If I execute the same query (with debugQuery=on) it gets parsed in
> > different ways on the two systems and I don't understand why.
>
> Look in solr.log. The full query, including parameters that are used
> but not on the URL, will be shown there. Provide that whole line from
> both versions.
>
> An example of the kind of line you need to find, with a very simple
> query, is below:
>
> 2019-06-05 15:50:23.691 INFO (qtp1264413185-43) [ x:foo]
> o.a.s.c.S.Request [foo] webapp=/solr path=/select
> params={q=*:*&_=1559749821933} hits=0 status=0 QTime=38
>
> If your index has multiple shards, there can be multiple lines. In that
> situation, we need the last one, which should be the main query itself
> rather than the subqueries.
>
> Thanks,
> Shawn
RE: query parsed in different ways in two identical solr instances
Hello, and thank you for your answer.
Attached you will find the two logs for the working solr1 server, and the
non-working solr-test server.
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
As for the European General Data Protection Regulation 2016/679 on the
protection of natural persons with regard to the processing of personal data,
we inform you that all the data we possess are object of treatment in the
respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may
ask for their correction, cancellation or you may oppose to their use by
written request sent by recorded delivery to The Microsoft Research –
University of Trento Centre for Computational and Systems Biology Scarl, Piazza
Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to
From: Shawn Heisey [apa...@elyograg.org]
Sent: 05 June 2019 17:52
To: solr-user@lucene.apache.org
Subject: Re: query parsed in different ways in two identical solr instances
On 6/5/2019 8:41 AM, Danilo Tomasoni wrote:
> Hello,
> I have two solr instances with exactly the same configuration.
> The only difference that i know is that the first (the working one, is solr
> 7.3.0,
> while the one that's not working is solr 7.3.1)
>
> If I execute the same query (with debugQuery=on) it gets parsed in different
> ways on the two systems and I don't understand why.
Look in solr.log. The full query, including parameters that are used
but not on the URL, will be shown there. Provide that whole line from
both versions.
An example of the kind of line you need to find, with a very simple
query, is below:
2019-06-05 15:50:23.691 INFO (qtp1264413185-43) [ x:foo]
o.a.s.c.S.Request [foo] webapp=/solr path=/select
params={q=*:*&_=1559749821933} hits=0 status=0 QTime=38
If your index has multiple shards, there can be multiple lines. In that
situation, we need the last one, which should be the main query itself
rather than the subqueries.
Thanks,
Shawn
2019-06-06 11:27:39.303 INFO (qtp2114889273-802) [ x:COSBIBioIndex] o.a.s.c.S.Request [COSBIBioIndex] webapp=/solr path=/select params={f.f2.qf=medline_chemical_terms+medline_mesh_terms=abstract_results^2+body_unnamed^1+titles^3+sponsors^1+keywords^10+abstracts^2+body_subjects^1+medline_mesh_terms^8+abstract_objective^2+body_nonstandard^1+body_materials_and_methods^1+body_results^1+annotation_terms^7+body_conclusions^1+body_ethics^1+abstract_methods^2+body_introduction^1+body_discussion^1+abstract_background^2+body_supplementary_materials^1+medline_chemical_terms^8+abstract_conclusions^2+body_funding^1+annotation_canonical_terms^7=on=*,+score,freq_secondary_ids_PUBMEDPMID12159614:termfreq(secondary_ids,+'PUBMEDPMID12159614'),freq_pmid_PUBMEDPMID12159614:termfreq(pmid,+'PUBMEDPMID12159614'),freq_doi_PUBMEDPMID12159614:termfreq(doi,+'PUBMEDPMID12159614'),freq_pmc_PUBMEDPMID12159614:termfreq(pmc,+'PUBMEDPMID12159614'),freq_id_PUBMEDPMID12159614:termfreq(id,+'PUBMEDPMID12159614'),freq_source_id_PUBMEDPMID12159614:termfreq(source_id,+'PUBMEDPMID12159614'),freq_manuscript_id_PUBMEDPMID12159614:termfreq(manuscript_id,+'PUBMEDPMID12159614'),freq_other_id_PUBMEDPMID12159614:termfreq(other_id,+'PUBMEDPMID12159614'),freq_publication_id_PUBMEDPMID12159614:termfreq(publication_id,+'PUBMEDPMID12159614'),freq_medline_mesh_terms_Models:termfreq(medline_mesh_terms,+'Models'),freq_medline_chemical_terms_Models:termfreq(medline_chemical_terms,+'Models'),freq_medline_mesh_terms_Molecular:termfreq(medline_mesh_terms,+'Molecular'),freq_medline_chemical_terms_Molecular:termfreq(medline_chemical_terms,+'Molecular'),freq_medline_mesh_terms_Models:termfreq(medline_mesh_terms,+'Models'),freq_medline_chemical_terms_Models:termfreq(medline_chemical_terms,+'Models'),freq_medline_mesh_terms_Theoretical:termfreq(medline_mesh_terms,+'Theoretical'),freq_medline_chemical_terms_Theoretical:termfreq(medline_chemical_terms,+'Theoretical'),freq_medline_mesh_terms_Models:termfreq(medline_mesh_terms,+'Models'),freq_medline_chemical_terms_Models:termfreq(medline_chemical_terms,+'Models'),freq_medline_mesh_terms_Statistical:termfreq(medline_mesh_terms,+'Statistical'),freq_medline_chemical_terms_Statistical:termfreq(medline_chemical_terms,+'Statistical'),freq_medline_mesh_terms_Models:termfreq(medline_mesh_terms,+'Models'),freq_medline_chemical_terms_Models:termfreq(medline_chemical_terms,+'Models'),freq_medline_mesh_terms_Immunological:termfreq(medline_mesh_terms,+'Immunological'),freq_medline_chemical_terms_Immunological:termfreq(medline_chemical_terms,+'Immunological'),freq_medline_mesh_terms_Molecular:termfreq(medline_mesh_terms,+'Molecular'),freq_medline_chemical_terms_Molecular:ter
Re: query parsed in different ways in two identical solr instances
On 6/5/2019 8:41 AM, Danilo Tomasoni wrote:
Hello,
I have two solr instances with exactly the same configuration.
The only difference that i know is that the first (the working one, is solr
7.3.0,
while the one that's not working is solr 7.3.1)
If I execute the same query (with debugQuery=on) it gets parsed in different
ways on the two systems and I don't understand why.
Look in solr.log. The full query, including parameters that are used
but not on the URL, will be shown there. Provide that whole line from
both versions.
An example of the kind of line you need to find, with a very simple
query, is below:
2019-06-05 15:50:23.691 INFO (qtp1264413185-43) [ x:foo]
o.a.s.c.S.Request [foo] webapp=/solr path=/select
params={q=*:*&_=1559749821933} hits=0 status=0 QTime=38
If your index has multiple shards, there can be multiple lines. In that
situation, we need the last one, which should be the main query itself
rather than the subqueries.
Thanks,
Shawn
query parsed in different ways in two identical solr instances
Hello,
I have two solr instances with exactly the same configuration.
The only difference that i know is that the first (the working one, is solr
7.3.0,
while the one that's not working is solr 7.3.1)
If I execute the same query (with debugQuery=on) it gets parsed in different
ways on the two systems and I don't understand why.
The query is
(f1:PUBMEDPMID12159614 AND (_query_:\"{!edismax qf='medline_chemical_terms
medline_mesh_terms' q.op=OR mm=1 v=$subquery1}\"))
and on solr 7.3.1 is parsedquery is
+(
DisjunctionMaxQuery((manuscript_id:PUBMEDPMID12159614 |
publication_id:PUBMEDPMID12159614
| pmc:PUBMEDPMID12159614 | id:PUBMEDPMID12159614 |
source_id:PUBMEDPMID12159614 | other_id:PUBMEDPMID12159614
| pmid:PUBMEDPMID12159614 | doi:PUBMEDPMID12159614)~0.1)
+(DisjunctionMaxQuery((medline_chemical_terms:\"model molecular\" |
medline_mesh_terms:\"model molecular\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model theoret\" |
medline_mesh_terms:\"model theoret\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model statist\" |
medline_mesh_terms:\"model statist\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model immunolog\" |
medline_mesh_terms:\"model immunolog\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"molecular dynam simul\" |
medline_mesh_terms:\"molecular dynam simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"molecular dock simul\" |
medline_mesh_terms:\"molecular dock simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"nonlinear dynam\" |
medline_mesh_terms:\"nonlinear dynam\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"numer analysi comput
assist\" | medline_mesh_terms:\"numer analysi comput assist\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"comput simul\" |
medline_mesh_terms:\"comput simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"comput biologi\" |
medline_mesh_terms:\"comput biologi\")~0.1)
)~1
)
while on solr 7.3.0 is
+(
+DisjunctionMaxQuery(
(manuscript_id:PUBMEDPMID12159614 | publication_id:PUBMEDPMID12159614 |
pmc:PUBMEDPMID12159614 | id:PUBMEDPMID12159614
| source_id:PUBMEDPMID12159614 | other_id:PUBMEDPMID12159614 |
pmid:PUBMEDPMID12159614 | secondary_ids:pubmedpmid12159614
| doi:PUBMEDPMID12159614)~0.1)
+(
+(
DisjunctionMaxQuery((medline_chemical_terms:\"model molecular\" |
medline_mesh_terms:\"model molecular\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model theoret\" |
medline_mesh_terms:\"model theoret\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model statist\" |
medline_mesh_terms:\"model statist\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"model immunolog\" |
medline_mesh_terms:\"model immunolog\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"molecular dynam
simul\" | medline_mesh_terms:\"molecular dynam simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"molecular dock
simul\" | medline_mesh_terms:\"molecular dock simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"nonlinear dynam\" |
medline_mesh_terms:\"nonlinear dynam\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"numer analysi comput
assist\" | medline_mesh_terms:\"numer analysi comput assist\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"comput simul\" |
medline_mesh_terms:\"comput simul\")~0.1)
DisjunctionMaxQuery((medline_chemical_terms:\"comput biologi\" |
medline_mesh_terms:\"comput biologi\")~0.1)
)~1
)
)
And the correct one is the latter.
Can someone explain me why this happens? the main query uses the edismax parser.
Thank you a very much for your help.
Danilo Tomasoni
Fondazione The Microsoft Research - University of Trento Centre for
Computational and Systems Biology (COSBI)
Piazza Manifattura 1, 38068 Rovereto (TN), Italy
tomas...@cosbi.eu
http://www.cosbi.eu
