Solr Grouping and Unique values

[Reinhardt, Nate]

I am trying to find a way to grab unique values based on a group. The idea 
would be to group by an id and then return that groups value.
Query params fl=valueIwant+myID&group=true&group.field=myId&q=:
"grouped": {
 "myID": {
 "matches": 7520236,
 "groups": [{
 "groupValue": "123456",
 "doclist": {
 "numFound": 6583,
 "start": 0,
 "docs": [{
 "myID": 123456,
 "valueIwant": "Hello World"
 }]
 }
 }
 ]
 }
}
This is fine but what I want to do is select the 'valueIwant' in a distinct 
way. The group.limit will return more values in the docs, but it wont be 
unique. Is there a way to restrict group.limit to only return unique fl values? 
With 6583 found for the above example. I would have to expand the limit to 6583 
then widdle it down by unique. This gets compounded when I have 700 unique ids 
that i want to group by with a total of 44m documents.
For example. If I do
fl=valueIwant+myID&group.limit=3&group=true&group.field=myId&q=:
 "grouped": {
 "myID": {
 "matches": 7520236,
 "groups": [{
 "groupValue": "123456",
 "doclist": {
 "numFound": 6583,
 "start": 0,
 "docs": [{
 "myID": 123456,
 "valueIwant": "Hello World"
 },
 {
 "myID": 123456,
 "valueIwant": "Hello World"
 }
 {
 "myID": 123456,
 "valueIwant": "Hello World123456"
 }]]
 }
 }
 ]
 }
 }
What I want is the docs to be unique against valueIwant like so
 "grouped": {
 "myID": {
 "matches": 7520236,
 "groups": [{
 "groupValue": "123456",
 "doclist": {
 "numFound": 6583,
 "start": 0,
 "docs": [{
 "myID": 123456,
 "valueIwant": "Hello World"
 },
 {
 "myID": 123456,
 "valueIwant": "Hello Planet"
 }
 {
 "myID": 123456,
 "valueIwant": "Hello World123456"
 }]]
 }
 }
 ]
 }
}
Is there a way to do this? I was looking at functions but couldnt find anything 
I needed.
Copy of this question can be found 
https://stackoverflow.com/questions/62679939/solr-grouping-and-unique-values


Thanks

Nate

Nested grouping

[Srinivas Kashyap]

Hi All,

I have below requirement for my business:

select?fl=*&fq=MODIFY_TS:[2020-06-23T18:30:00Z TO *]&fq=PHY_KEY2: "HQ010699" OR 
PHY_KEY2: "HQ010377" OR PHY_KEY2: "HQ010396" OR PHY_KEY2: "HQ010399" OR 
PHY_KEY2: "HQ010404" OR PHY_KEY2: "HQ010419" OR PHY_KEY2: "HQ010426" OR 
PHY_KEY2: "HQ010452" OR PHY_KEY2: "HQ010463" OR PHY_KEY2: "HQ010466" OR 
PHY_KEY2: "HQ010469" OR PHY_KEY2: "HQ010476" OR PHY_KEY2: "HQ010480" OR 
PHY_KEY2: "HQ010481" OR PHY_KEY2: "HQ010496" OR PHY_KEY2: "HQ010500" OR 
PHY_KEY2: "HQ010501" OR PHY_KEY2: "HQ010502" OR PHY_KEY2: "HQ010503" OR 
PHY_KEY2: "HQ010504"

Above query lists all the changes mentioned for 20 documents.

If I add below to query:

group=true&group.field=PHY_KEY2&group.ngroups=true

Below is the response:

"grouped":{
"PHY_KEY2":{
  "matches":23,
  "ngroups":4,
  "groups":[{
  "groupValue":"HQ010500",
  "doclist":{"numFound":3,"start":0,"docs":[
  {
"PHY_KEY2":"HQ010500"}]
  }},
{
  "groupValue":"HQ010399",
  "doclist":{"numFound":4,"start":0,"docs":[
  {
"PHY_KEY2":"HQ010399"}]
  }},
{
  "groupValue":"HQ010377",
  "doclist":{"numFound":8,"start":0,"docs":[
  {
"PHY_KEY2":"HQ010377"}]
  }},
{
  "groupValue":"HQ010699",
  "doclist":{"numFound":8,"start":0,"docs":[
  {
"PHY_KEY2":"HQ010699"}]
  }}]}}}

Take the case of last entry, HQ010699. It says numFound=8 (8 docs). But all of 
these 8 documents have the same value for field called TRACK_ID.  So can I 
group again on the TRACK_ID to get the count as 1.


Thanks and Regards,
Srinivas Kashyap


DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Grouping and Learning To Rank (SOLR-8776)

[Webster Homer]

My company is very interested in using Learning To Rank in our product search. 
The problem we face is that our product search groups its results and that does 
not work with LTR.
https://issues.apache.org/jira/browse/SOLR-8776

Is there any traction to getting the SOLR-8776 patch into the main branch? 
Seems like this would be useful to a lot of people



This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, you 
must not copy this message or attachment or disclose the contents to any other 
person. If you have received this transmission in error, please notify the 
sender immediately and delete the message and any attachment from your system. 
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept 
liability for any omissions or errors in this message which may arise as a 
result of E-Mail-transmission or for damages resulting from any unauthorized 
changes of the content of this message and any attachment thereto. Merck KGaA, 
Darmstadt, Germany and any of its subsidiaries do not guarantee that this 
message is free of viruses and does not accept liability for any damages caused 
by any virus transmitted therewith.



Click http://www.merckgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: Solr grouping with offset

[Saurabh Sharma]

Hi,

Yes. I meant facet.mincount only.


Thanks
Saurabh

On Fri, Feb 14, 2020, 8:51 PM Vadim Ivanov <
vadim.iva...@spb.ntk-intourist.ru> wrote:

> group.mincount ? Never heard of it. It exists?
> May be you have in mind facet.mincount and second approach mentioned
> earlier:
>
> > > > > Next approach was to use facet first with facet.mincount=3, then
> > > > > find docs ids by every facet result  and then delete docs by id.
> > > > > That way seems to me  too complicated for the task.
>
> > -Original Message-
> > From: Saurabh Sharma [mailto:saurabh.infoe...@gmail.com]
> > Sent: Friday, February 14, 2020 4:36 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr grouping with offset
> >
> > Hi,
> >
> > If you want to sort on your field and want to put a count restriction
> too then
> > you have to use mincount. That seems to be best approach for your
> > problem.
> >
> > Thanks
> > Saurabh
> >
> > On Fri, Feb 14, 2020, 6:24 PM Vadim Ivanov < vadim.iva...@spb.ntk-
> > intourist.ru> wrote:
> >
> > > Example of gtouping with empty groups in results:
> > > Filed1 = rr_group, field2 = rr_updatedate Problem is that I have tens
> > > of million groups in result and only several thousand with  "numFound"
> > > >2
> > >
> > > "params":{
> > >   "q":"*:* ",
> > >   "group.sort":"rr_updatedate desc ",
> > >   "group.limit":"-1",
> > >   "fl":"rr_group,rr_adl,rr_createdate,rr_calctaskkey ",
> > >   "group.offset":"2",
> > >   "wt":"json",
> > >   "group.field":"rr_group",
> > >   "group":"true"}},
> > >   "grouped":{
> > > "rr_group":{
> > >   "matches":41475082,
> > >   "groups":[{
> > >   "groupValue":"164370:20200707:23:251",
> > >   "doclist":{"numFound":1,"start":2,"docs":[]
> > >   }},
> > > {
> > >   "groupValue":"163942:20200708:22:251",
> > >   "doclist":{"numFound":1,"start":2,"docs":[]
> > >   }},
> > > {
> > >   "groupValue":"163943:20200708:22:251",
> > >   "doclist":{"numFound":1,"start":2,"docs":[]
> > >   }},
> > > {
> > >   "groupValue":"164355:20200708:22:251",
> > >   "doclist":{"numFound":1,"start":2,"docs":[]
> > >
> > > > -Original Message-
> > > > From: Paras Lehana [mailto:paras.leh...@indiamart.com]
> > > > Sent: Friday, February 14, 2020 3:37 PM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: Solr grouping with offset
> > > >
> > > > It would be better if you give us an example.
> > > >
> > > > On Fri, 14 Feb 2020 at 17:20, Vadim Ivanov
> > > >  wrote:
> > > >
> > > > > Hello guys!
> > > > > I need an advise. My task is to delete some documents in
> collection.
> > > > > Del algorithm is following:
> > > > > Group docs by field1  with sort by field2 and delete every 3 and
> > > > > following occurrences in every group.
> > > > > Unfortunately I didn't find easy way to do so.
> > > > > Closest approach was to use group.offset = 2, but  result set is
> > > > > polluted with empty groups with no documents (they have less then
> > > > > 3
> > > docs
> > > > in group).
> > > > > May be I'm missing smth and there is way not to receive empty
> > > > > groups in results?
> > > > > Next approach was to use facet first with facet.mincount=3, then
> > > > > find docs ids by every facet result  and then delete docs by id.
> > > > > That way seems to me  too complicated for the task.
> > > > > What's the best use case for the task?
> > > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > Regards,
> > > >
> > > > *Paras Lehana* [65871]
> > > > Development Engineer, *Auto-Suggest*, IndiaMART InterMESH Ltd,
> > > >
> > > > 11th Floor, Tower 2, Assotech Business Cresterra, Plot No. 22,
> > > > Sector
> > > 135,
> > > > Noida, Uttar Pradesh, India 201305
> > > >
> > > > Mob.: +91-9560911996
> > > > Work: 0120-4056700 | Extn:
> > > > *11096*
> > > >
> > > > --
> > > > *
> > > > *
> > > >
> > > >  <https://www.facebook.com/IndiaMART/videos/578196442936091/>
> > >
> > >
>
>

RE: Solr grouping with offset

[Vadim Ivanov]

group.mincount ? Never heard of it. It exists?
May be you have in mind facet.mincount and second approach mentioned earlier:

> > > > Next approach was to use facet first with facet.mincount=3, then
> > > > find docs ids by every facet result  and then delete docs by id.
> > > > That way seems to me  too complicated for the task.

> -Original Message-
> From: Saurabh Sharma [mailto:saurabh.infoe...@gmail.com]
> Sent: Friday, February 14, 2020 4:36 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr grouping with offset
> 
> Hi,
> 
> If you want to sort on your field and want to put a count restriction too then
> you have to use mincount. That seems to be best approach for your
> problem.
> 
> Thanks
> Saurabh
> 
> On Fri, Feb 14, 2020, 6:24 PM Vadim Ivanov < vadim.iva...@spb.ntk-
> intourist.ru> wrote:
> 
> > Example of gtouping with empty groups in results:
> > Filed1 = rr_group, field2 = rr_updatedate Problem is that I have tens
> > of million groups in result and only several thousand with  "numFound"
> > >2
> >
> > "params":{
> >   "q":"*:* ",
> >   "group.sort":"rr_updatedate desc ",
> >   "group.limit":"-1",
> >   "fl":"rr_group,rr_adl,rr_createdate,rr_calctaskkey ",
> >   "group.offset":"2",
> >   "wt":"json",
> >   "group.field":"rr_group",
> >   "group":"true"}},
> >   "grouped":{
> > "rr_group":{
> >   "matches":41475082,
> >   "groups":[{
> >   "groupValue":"164370:20200707:23:251",
> >   "doclist":{"numFound":1,"start":2,"docs":[]
> >   }},
> > {
> >   "groupValue":"163942:20200708:22:251",
> >   "doclist":{"numFound":1,"start":2,"docs":[]
> >   }},
> > {
> >   "groupValue":"163943:20200708:22:251",
> >   "doclist":{"numFound":1,"start":2,"docs":[]
> >   }},
> > {
> >   "groupValue":"164355:20200708:22:251",
> >   "doclist":{"numFound":1,"start":2,"docs":[]
> >
> > > -Original Message-
> > > From: Paras Lehana [mailto:paras.leh...@indiamart.com]
> > > Sent: Friday, February 14, 2020 3:37 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Solr grouping with offset
> > >
> > > It would be better if you give us an example.
> > >
> > > On Fri, 14 Feb 2020 at 17:20, Vadim Ivanov
> > >  wrote:
> > >
> > > > Hello guys!
> > > > I need an advise. My task is to delete some documents in collection.
> > > > Del algorithm is following:
> > > > Group docs by field1  with sort by field2 and delete every 3 and
> > > > following occurrences in every group.
> > > > Unfortunately I didn't find easy way to do so.
> > > > Closest approach was to use group.offset = 2, but  result set is
> > > > polluted with empty groups with no documents (they have less then
> > > > 3
> > docs
> > > in group).
> > > > May be I'm missing smth and there is way not to receive empty
> > > > groups in results?
> > > > Next approach was to use facet first with facet.mincount=3, then
> > > > find docs ids by every facet result  and then delete docs by id.
> > > > That way seems to me  too complicated for the task.
> > > > What's the best use case for the task?
> > > >
> > >
> > >
> > > --
> > > --
> > > Regards,
> > >
> > > *Paras Lehana* [65871]
> > > Development Engineer, *Auto-Suggest*, IndiaMART InterMESH Ltd,
> > >
> > > 11th Floor, Tower 2, Assotech Business Cresterra, Plot No. 22,
> > > Sector
> > 135,
> > > Noida, Uttar Pradesh, India 201305
> > >
> > > Mob.: +91-9560911996
> > > Work: 0120-4056700 | Extn:
> > > *11096*
> > >
> > > --
> > > *
> > > *
> > >
> > >  <https://www.facebook.com/IndiaMART/videos/578196442936091/>
> >
> >

Re: Solr grouping with offset

[Saurabh Sharma]

Hi,

If you want to sort on your field and want to put a count restriction too
then you have to use mincount. That seems to be best approach for your
problem.

Thanks
Saurabh

On Fri, Feb 14, 2020, 6:24 PM Vadim Ivanov <
vadim.iva...@spb.ntk-intourist.ru> wrote:

> Example of gtouping with empty groups in results:
> Filed1 = rr_group, field2 = rr_updatedate
> Problem is that I have tens of million groups in result and only several
> thousand with  "numFound" >2
>
> "params":{
>   "q":"*:* ",
>   "group.sort":"rr_updatedate desc ",
>   "group.limit":"-1",
>   "fl":"rr_group,rr_adl,rr_createdate,rr_calctaskkey ",
>   "group.offset":"2",
>   "wt":"json",
>   "group.field":"rr_group",
>   "group":"true"}},
>   "grouped":{
> "rr_group":{
>   "matches":41475082,
>   "groups":[{
>   "groupValue":"164370:20200707:23:251",
>   "doclist":{"numFound":1,"start":2,"docs":[]
>   }},
> {
>   "groupValue":"163942:20200708:22:251",
>   "doclist":{"numFound":1,"start":2,"docs":[]
>   }},
> {
>   "groupValue":"163943:20200708:22:251",
>   "doclist":{"numFound":1,"start":2,"docs":[]
>   }},
> {
>   "groupValue":"164355:20200708:22:251",
>   "doclist":{"numFound":1,"start":2,"docs":[]
>
> > -Original Message-
> > From: Paras Lehana [mailto:paras.leh...@indiamart.com]
> > Sent: Friday, February 14, 2020 3:37 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr grouping with offset
> >
> > It would be better if you give us an example.
> >
> > On Fri, 14 Feb 2020 at 17:20, Vadim Ivanov
> >  wrote:
> >
> > > Hello guys!
> > > I need an advise. My task is to delete some documents in collection.
> > > Del algorithm is following:
> > > Group docs by field1  with sort by field2 and delete every 3 and
> > > following occurrences in every group.
> > > Unfortunately I didn't find easy way to do so.
> > > Closest approach was to use group.offset = 2, but  result set is
> > > polluted with empty groups with no documents (they have less then 3
> docs
> > in group).
> > > May be I'm missing smth and there is way not to receive empty groups
> > > in results?
> > > Next approach was to use facet first with facet.mincount=3, then find
> > > docs ids by every facet result  and then delete docs by id.
> > > That way seems to me  too complicated for the task.
> > > What's the best use case for the task?
> > >
> >
> >
> > --
> > --
> > Regards,
> >
> > *Paras Lehana* [65871]
> > Development Engineer, *Auto-Suggest*,
> > IndiaMART InterMESH Ltd,
> >
> > 11th Floor, Tower 2, Assotech Business Cresterra, Plot No. 22, Sector
> 135,
> > Noida, Uttar Pradesh, India 201305
> >
> > Mob.: +91-9560911996
> > Work: 0120-4056700 | Extn:
> > *11096*
> >
> > --
> > *
> > *
> >
> >  <https://www.facebook.com/IndiaMART/videos/578196442936091/>
>
>

RE: Solr grouping with offset

[Vadim Ivanov]

Example of gtouping with empty groups in results:
Filed1 = rr_group, field2 = rr_updatedate
Problem is that I have tens of million groups in result and only several 
thousand with  "numFound" >2
   
"params":{
  "q":"*:* ",
  "group.sort":"rr_updatedate desc ",
  "group.limit":"-1",
  "fl":"rr_group,rr_adl,rr_createdate,rr_calctaskkey ",
  "group.offset":"2",
  "wt":"json",
  "group.field":"rr_group",
  "group":"true"}},
  "grouped":{
"rr_group":{
  "matches":41475082,
  "groups":[{
  "groupValue":"164370:20200707:23:251",
  "doclist":{"numFound":1,"start":2,"docs":[]
  }},
{
  "groupValue":"163942:20200708:22:251",
  "doclist":{"numFound":1,"start":2,"docs":[]
  }},
{
  "groupValue":"163943:20200708:22:251",
  "doclist":{"numFound":1,"start":2,"docs":[]
  }},
{
  "groupValue":"164355:20200708:22:251",
  "doclist":{"numFound":1,"start":2,"docs":[]

> -Original Message-
> From: Paras Lehana [mailto:paras.leh...@indiamart.com]
> Sent: Friday, February 14, 2020 3:37 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr grouping with offset
> 
> It would be better if you give us an example.
> 
> On Fri, 14 Feb 2020 at 17:20, Vadim Ivanov
>  wrote:
> 
> > Hello guys!
> > I need an advise. My task is to delete some documents in collection.
> > Del algorithm is following:
> > Group docs by field1  with sort by field2 and delete every 3 and
> > following occurrences in every group.
> > Unfortunately I didn't find easy way to do so.
> > Closest approach was to use group.offset = 2, but  result set is
> > polluted with empty groups with no documents (they have less then 3 docs
> in group).
> > May be I'm missing smth and there is way not to receive empty groups
> > in results?
> > Next approach was to use facet first with facet.mincount=3, then find
> > docs ids by every facet result  and then delete docs by id.
> > That way seems to me  too complicated for the task.
> > What's the best use case for the task?
> >
> 
> 
> --
> --
> Regards,
> 
> *Paras Lehana* [65871]
> Development Engineer, *Auto-Suggest*,
> IndiaMART InterMESH Ltd,
> 
> 11th Floor, Tower 2, Assotech Business Cresterra, Plot No. 22, Sector 135,
> Noida, Uttar Pradesh, India 201305
> 
> Mob.: +91-9560911996
> Work: 0120-4056700 | Extn:
> *11096*
> 
> --
> *
> *
> 
>  <https://www.facebook.com/IndiaMART/videos/578196442936091/>

Re: Solr grouping with offset

[Paras Lehana]

It would be better if you give us an example.

On Fri, 14 Feb 2020 at 17:20, Vadim Ivanov
 wrote:

> Hello guys!
> I need an advise. My task is to delete some documents in collection.
> Del algorithm is following:
> Group docs by field1  with sort by field2 and delete every 3 and following
> occurrences in every group.
> Unfortunately I didn't find easy way to do so.
> Closest approach was to use group.offset = 2, but  result set is polluted
> with empty groups with no documents (they have less then 3 docs in group).
> May be I'm missing smth and there is way not to receive empty groups in
> results?
> Next approach was to use facet first with facet.mincount=3, then find docs
> ids by every facet result  and then delete docs by id.
> That way seems to me  too complicated for the task.
> What's the best use case for the task?
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, *Auto-Suggest*,
IndiaMART InterMESH Ltd,

11th Floor, Tower 2, Assotech Business Cresterra,
Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305

Mob.: +91-9560911996
Work: 0120-4056700 | Extn:
*11096*

-- 
*
*

 

Solr grouping with offset

[Vadim Ivanov]

Hello guys!
I need an advise. My task is to delete some documents in collection.
Del algorithm is following:
Group docs by field1  with sort by field2 and delete every 3 and following 
occurrences in every group.
Unfortunately I didn't find easy way to do so.
Closest approach was to use group.offset = 2, but  result set is polluted with 
empty groups with no documents (they have less then 3 docs in group).
May be I'm missing smth and there is way not to receive empty groups in results?
Next approach was to use facet first with facet.mincount=3, then find docs ids 
by every facet result  and then delete docs by id.
That way seems to me  too complicated for the task.
What's the best use case for the task?

Re: Grouping and sorting Together

[Paras Lehana]

Hi Neo,

Please mention the expected result as well. Do you want to sort "item sold"
group wise or result wise? Use sort
<https://lucene.apache.org/solr/guide/8_3/common-query-parameters.html#sort-parameter>
for the former while group.sort
<https://lucene.apache.org/solr/guide/8_3/result-grouping.html#grouping-parameters>
for the later. Describe your expected result set if I'm not answering
your question.

On Thu, 14 Nov 2019 at 19:35, neotorand  wrote:

> Hi List
> I need your help to resolve a problem for which i had been struggling for
> days.
> Lets take an example of Shoes which are grouped on basis of size and Price
>
> With first group as size and price as "7 and 7000" i have 2 documents as
> below
>
> {id:1,color:blue,item sold:10}
> {id:5,price:yellow,item sold:1}
>
>
> with second group as size and price as "8 and 8000"  i have 2 documents as
> below
>
> {id:2,color:blue,item sold:3}
> {id:3,price:yellow,item sold:5}
>
> Now i want to sort the records based on item sold.
> How I should look at  the problem.should i remove grouping and sort result
> and show.I m asking this as u can see first group has item with item sold
> as
> 10,1 and second group as 3 and 5.
> What approach i should have to look at the problem
>
> Regards
> Neo
>
>
>
>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.

Grouping and sorting Together

[neotorand]

Hi List
I need your help to resolve a problem for which i had been struggling for
days.
Lets take an example of Shoes which are grouped on basis of size and Price

With first group as size and price as "7 and 7000" i have 2 documents as
below

{id:1,color:blue,item sold:10}
{id:5,price:yellow,item sold:1}


with second group as size and price as "8 and 8000"  i have 2 documents as
below

{id:2,color:blue,item sold:3}
{id:3,price:yellow,item sold:5}

Now i want to sort the records based on item sold.
How I should look at  the problem.should i remove grouping and sort result
and show.I m asking this as u can see first group has item with item sold as
10,1 and second group as 3 and 5.
What approach i should have to look at the problem

Regards
Neo







--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Grouping facet count is not working with JSON Faceting

[Ganesan, VinothKumar]

Hi,

I am trying to use JSON faceting in SOLR 7.3 along with grouping documents and 
noticed that group.facet=true is not working with json.facet but it works with 
flat faceting.

Query Format I used: 
http://localhost:8983/solr/Collection/select?q=*:*&facet=true&facet.mincount=2&json.facet={sc_uf_facetfield:{type:terms,limit:-1,sort:{index:asc},field:facetfield}}&group=false&group.ngroups=false&group.format=grouped&group.limit=1&group.field=groupid&group.facet=true<http://localhost:8983/solr/Collection/select?q=*:*&facet=true&facet.mincount=2&json.facet=%7bsc_uf_facetfield:%7btype:terms,limit:-1,sort:%7bindex:asc%7d,field:facetfield%7d%7d&group=false&group.ngroups=false&group.format=grouped&group.limit=1&group.field=groupid&group.facet=true>

I have 2 documents in my index and both have the same value for facetfield. I 
tried this with clustered and single SOLR node.

Expected

Actual

"sc_uf_facetfield": {
"buckets": [
{
"val": "facet value",
   "count": 1
}
]
}

"sc_uf_facetfield": {
"buckets": [
{
"val": "facet value",
"count": 2
}
]
}


Can you please let me know whether this is a known SOLR bug or my SOLR query is 
wrong?


Thanks,
Vinoth

Re: Softer version of grouping and/or filter query

[Edward Ribeiro]

Cool! Paraphrasing 'Solr in Action' book: edismax is the query parser to
use when dealing with users' queries. It has a lot of customization options
and is more resilient to ill-formed queries than lucene-parser. Whenever
possible, take some time to dig deeper into those. :)

Regards,
Edward

On Fri, May 10, 2019 at 6:09 PM Doug Reeder  wrote:

> Thanks much!  I dropped price from the fq term, changed to an edismax
> parser, and boosted with
> bq=price:[150+TO+*]^100
>
>
>
> On Thu, May 9, 2019 at 7:21 AM Edward Ribeiro 
> wrote:
>
> > Em qua, 8 de mai de 2019 18:56, Doug Reeder 
> > escreveu:
> >
> > >
> > > Similarly, we have a filter query that only returns products over $150:
> > > fq=price:[150+TO+*]
> > >
> > > Can this be changed to a q or qf parameter where products less than
> $150
> > > have score less than any product priced $150 or more? (A price higher
> > than
> > > $150 should not increase the score.)
> > >
> >
> > If you are using edismax then you could use boost function. Maybe
> something
> > along those: bf=if(lt(price, 150), 0.5, 100)
> >
> > Your fq already filters out documents with prices less than 150. Using a
> > boost (function/query) will retrieve back docs with prices less than 150,
> > but probably with smaller scores.
> >
> > Edward
> >
> > >
> >
>

Re: Softer version of grouping and/or filter query

[Doug Reeder]

Thanks much!  I dropped price from the fq term, changed to an edismax
parser, and boosted with
bq=price:[150+TO+*]^100



On Thu, May 9, 2019 at 7:21 AM Edward Ribeiro 
wrote:

> Em qua, 8 de mai de 2019 18:56, Doug Reeder 
> escreveu:
>
> >
> > Similarly, we have a filter query that only returns products over $150:
> > fq=price:[150+TO+*]
> >
> > Can this be changed to a q or qf parameter where products less than $150
> > have score less than any product priced $150 or more? (A price higher
> than
> > $150 should not increase the score.)
> >
>
> If you are using edismax then you could use boost function. Maybe something
> along those: bf=if(lt(price, 150), 0.5, 100)
>
> Your fq already filters out documents with prices less than 150. Using a
> boost (function/query) will retrieve back docs with prices less than 150,
> but probably with smaller scores.
>
> Edward
>
> >
>

Re: Softer version of grouping and/or filter query

[Edward Ribeiro]

Em qua, 8 de mai de 2019 18:56, Doug Reeder 
escreveu:

>
> Similarly, we have a filter query that only returns products over $150:
> fq=price:[150+TO+*]
>
> Can this be changed to a q or qf parameter where products less than $150
> have score less than any product priced $150 or more? (A price higher than
> $150 should not increase the score.)
>

If you are using edismax then you could use boost function. Maybe something
along those: bf=if(lt(price, 150), 0.5, 100)

Your fq already filters out documents with prices less than 150. Using a
boost (function/query) will retrieve back docs with prices less than 150,
but probably with smaller scores.

Edward

>

Re: Softer version of grouping and/or filter query

[Emir Arnautović]

Hi Doug,
It seems to me that you’ve found a way to increase score for those that are 
within selected price range, but “A price higher than $150 should not increase 
the score”. I’ll just remind you that scores in Solr are relevant to query and 
that you cannot do much other than sorting on it so it should not matter much 
if you boost the one that you like more or decrease score for those that are 
not your first choice.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 8 May 2019, at 23:56, Doug Reeder  wrote:
> 
> We have a query to return products related to a given product. To give some
> variety to the results, we group by vendor:
> group=true&group.main=true&group.field=merchantId
> 
> We need at least four results to display. Unfortunately, some categories
> don't have a lot of products, and grouping takes us (say) from five results
> to three.
> 
> Can I "soften" the grouping, so other products by the same vendor will
> appear in the results, but with much lower score?
> 
> 
> Similarly, we have a filter query that only returns products over $150:
> fq=price:[150+TO+*]
> 
> Can this be changed to a q or qf parameter where products less than $150
> have score less than any product priced $150 or more? (A price higher than
> $150 should not increase the score.)

Softer version of grouping and/or filter query

[Doug Reeder]

We have a query to return products related to a given product. To give some
variety to the results, we group by vendor:
group=true&group.main=true&group.field=merchantId

We need at least four results to display. Unfortunately, some categories
don't have a lot of products, and grouping takes us (say) from five results
to three.

Can I "soften" the grouping, so other products by the same vendor will
appear in the results, but with much lower score?


Similarly, we have a filter query that only returns products over $150:
fq=price:[150+TO+*]

Can this be changed to a q or qf parameter where products less than $150
have score less than any product priced $150 or more? (A price higher than
$150 should not increase the score.)

Re: Interesting Grouping/Facet issue

[Shawn Heisey]

On 4/9/2019 7:03 AM, Erie Data Systems wrote:

Solr 8.0.0, I have a HASHTAG string field I am trying to facet on to get
the most popular hashtags (top 100) across many sources. (SITE field is
string)

/select?facet.field=hashtag&facet=on&rows=0&q=%2Bhashtag:*%20%2BDT:[" .
date('Y-m-d') . "T00:00:00Z+TO+" . date('Y-m-d')  .
"T23:59:59Z]&facet.limit=100&facet.mincount=1&facet.method=fc

It works but not to what I feel should happen... For example if one site
has 1000 rows on todays date and they all have a HASHTAG in common, that
HASHTAG automatically rises to the top simply because one SITE has 1000
pages with the same HASHTAG.


That is exactly what faceting is designed to do.  It is behaving exactly 
as designed.



Is there a way to get a better more even distribution of top HASHTAGS for a
given date, ie facet. ..by a grouping or distinct or filter of some sort?
Im more interesting in knowing if a HASHTAG is used frequently among SITEs,
not just one one.


If you use pivot facets, first on the field you want to classify on, 
then on HASHTAG, that MIGHT get you what you want.


You could also try running many different facet queries, each one with a 
specific query and/or filter that achieves the results you want.


FYI:  Including "hashtag:*" in your query makes it a wildcard query. 
This is most likely VERY slow.  If you are trying to match all possible 
values in the hashtag field, then take it out, it's unnecessary.  If you 
are trying to match only documents where hashtag contains a value, then 
replace it with this for a performance improvement:


hashtag:[* TO *]

Range queries are almost always faster than wildcards.

Thanks,
Shawn

Interesting Grouping/Facet issue

[Erie Data Systems]

Solr 8.0.0, I have a HASHTAG string field I am trying to facet on to get
the most popular hashtags (top 100) across many sources. (SITE field is
string)

/select?facet.field=hashtag&facet=on&rows=0&q=%2Bhashtag:*%20%2BDT:[" .
date('Y-m-d') . "T00:00:00Z+TO+" . date('Y-m-d')  .
"T23:59:59Z]&facet.limit=100&facet.mincount=1&facet.method=fc

It works but not to what I feel should happen... For example if one site
has 1000 rows on todays date and they all have a HASHTAG in common, that
HASHTAG automatically rises to the top simply because one SITE has 1000
pages with the same HASHTAG.

Is there a way to get a better more even distribution of top HASHTAGS for a
given date, ie facet. ..by a grouping or distinct or filter of some sort?
Im more interesting in knowing if a HASHTAG is used frequently among SITEs,
not just one one.

Hope this makes sense... any recommendations welcomed.

Thank you in advance,
-Craig

Re: Unexpected docvalues type SORTED_NUMERIC Exception when grouping by a PointField facet

[Erick Erickson]

Looks like: https://issues.apache.org/jira/browse/SOLR-11728

> On Apr 3, 2019, at 1:09 AM, JiaJun Zhu  wrote:
> 
> Hello,
> 
> 
> I got an "Unexpected docvalues type SORTED_NUMERIC" exception when I perform 
> group facet on an IntPointField. Debugging into the source code, the cause is 
> that internally the docvalue type for PointField is "NUMERIC" (single value) 
> or "SORTED_NUMERIC" (multi value), while the TermGroupFacetCollector class 
> requires the facet field must have a "SORTED" or "SOTRTED_SET" docvalue type: 
> https://github.com/apache/lucene-solr/blob/2480b74887eff01f729d62a57b415d772f947c91/lucene/grouping/src/java/org/apache/lucene/search/grouping/TermGroupFacetCollector.java#L313
> 
> When I change schema for all int field to TrieIntField, the group facet then 
> work. Since internally the docvalue type for TrieField is SORTED (single 
> value) or SORTED_SET (multi value).
> 
> Regarding that the "TrieField" is depreciated in Solr7, can someone help on 
> this grouping facet issue for PointField. I also commented this issue in 
> SOLR-7495.
> 
> 
> Thanks.
> 
> 
> 
> Best regards,
> 
> JiaJun
> Manager Technology
> Alexander Street, a ProQuest Company
> No. 201 NingXia Road, Room 6J Shanghai China P.R.
> 200063

Unexpected docvalues type SORTED_NUMERIC Exception when grouping by a PointField facet

[JiaJun Zhu]

Hello,


I got an "Unexpected docvalues type SORTED_NUMERIC" exception when I perform 
group facet on an IntPointField. Debugging into the source code, the cause is 
that internally the docvalue type for PointField is "NUMERIC" (single value) or 
"SORTED_NUMERIC" (multi value), while the TermGroupFacetCollector class 
requires the facet field must have a "SORTED" or "SOTRTED_SET" docvalue type: 
https://github.com/apache/lucene-solr/blob/2480b74887eff01f729d62a57b415d772f947c91/lucene/grouping/src/java/org/apache/lucene/search/grouping/TermGroupFacetCollector.java#L313

When I change schema for all int field to TrieIntField, the group facet then 
work. Since internally the docvalue type for TrieField is SORTED (single value) 
or SORTED_SET (multi value).

Regarding that the "TrieField" is depreciated in Solr7, can someone help on 
this grouping facet issue for PointField. I also commented this issue in 
SOLR-7495.


Thanks.



Best regards,

JiaJun
Manager Technology
Alexander Street, a ProQuest Company
No. 201 NingXia Road, Room 6J Shanghai China P.R.
200063

solr grouping

[swap]

I have document in solr as mentioned below

{
"event_name":"product viewed",

"event_property":["category","product_name","product_code","price","brand","color","discount","is_new_visitor"],
"event_value":["category-sunglasses","product_name-david blake grey
sunglasses","product_code-lcsgdb364x1880gryx""price-590","brand-david
blake","color-grey","discount-70"],
"session_id":"mf154521205475440",
"company_id":"31",
"created":1545212153,
"email":"z...@gmail.com",
"name":""
},
{
"event_name":"product viewed",

"event_property":["category","product_name","product_code","price","brand","color","discount","is_new_visitor"],
"event_value":["category-sunglasses","product_name-david blake grey
sunglasses","product_code-lcsgdb364x1880gryx""price-590","brand-david
blake","color-grey","discount-70"],
"session_id":"mf154521205475440",
"company_id":"31",
"created":1545212153,
"email":"y...@gmail.com",
"name":""
}

i need to query and group email with filtered query fq as mentioned below 

http://solr-url/solr/solr-core/select?q=:&fq=((event_name:"product+viewed"+AND+event_property:"product_name"+AND+event_value:"category-sunglasses")+AND+(event_name:"add+to+cart"+AND+event_property:"product_name"+AND+event_value:"category-sunglasses"))&group.limit=1&group.ngroups=true&group=true&group.field=email

i need filter query
(event_name:"product+viewed"+AND+event_property:"product_name"+AND+event_value:"category-sunglasses")
with "AND" on
event_name:"add+to+cart"+AND+event_property:"product_name"+AND+event_value:"category-sunglasses")

In response i am not getting the email e.g user who as performed both event
e.g "product viewed" and "add to cart".

document have email and activity perform by user in event key.now i need to
make request to group email to find unique email using filter query.i have
used the query mentioned below







--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Pagination with grouping in solr

[swap]

Document structure of solr document is as mentioned below now i need to get
the document having event_name="product view" and group it by email so that
email is not duplicate.Now on listing the email how may paginate the unique
email.As the query return total number of document not the count of groups

"docs":[ { "id":"1", "email":"xxx...@gmail.com", "gender":"M",
"location":["yyy"], "created":123444, "event_name":"product viewed",
"event_property":"product", "event_value":"sun glassed",
"version":1617201602734587904, "location_str":[""] }, { "id":"4",
"email":"xxx...@gmail.com", "gender":"F", "location":[""],
"created":123447, "event_name":"Add To Cart", "event_property":"Name",
"event_value":"sun glasses", "version":1617202784870858752,
"location_str":[""] }, { "id":"5", "email":"xxx...@gmail.com",
"gender":"M", "location":["k"], "created":123464, "event_name":"Product
Clicked", "event_property":"Category", "event_value":"Contact Lens",
"version":1617202784871907328, "location_str":["l"] } ]



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: distributed grouping by date

[Erick Erickson]

Looks like: https://issues.apache.org/jira/browse/SOLR-11086
On Tue, Nov 6, 2018 at 8:19 AM Tomáš Hampl  wrote:
>
> Hi,
>
> i have error while running grouping query by date in collection with 5
> shards. When i try same query on collection with only one shard everything
> works.
>
> *query:*
>
> /solr/search_cz/select?q=*:*&group=true&group.field=odjezd
>
> *part of schema.xml*
>
>  positionIncrementGap="0"/>
>
> ...
>
> 
>
> *collection create *
>
> /solr/admin/collections\?action\=CREATE\&autoAddReplicas\=true\&collection.configName\=search_cz\&maxShardsPerNode\=5\&name\=search_cz\&numShards\=5\&replicationFactor\=1\&router.field\=routing_key\&
> router.name\=compositeId\&rule\=shard:\*,replica:\<2,node:\*
>
>
> *stacktrace:*
>
> org.apache.solr.common.SolrException: Invalid Date String:'Wed Nov 21
> 12:45:00 UTC 2018'
> at 
> org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:247)
> at 
> org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:226)
> at 
> org.apache.solr.schema.DatePointField.toNativeType(DatePointField.java:113)
> at 
> org.apache.solr.schema.DatePointField.readableToIndexed(DatePointField.java:184)
> at 
> org.apache.solr.search.grouping.distributed.command.GroupConverter.fromMutable(GroupConverter.java:57)
> at 
> org.apache.solr.search.grouping.distributed.command.SearchGroupsFieldCommand.result(SearchGroupsFieldCommand.java:128)
> at 
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:55)
> at 
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:37)
> at 
> org.apache.solr.search.grouping.CommandHandler.processResult(CommandHandler.java:208)
> at 
> org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchFirstPhase(QueryComponent.java:1282)
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:360)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at 
>

distributed grouping by date

[Tomáš Hampl]

Hi,

i have error while running grouping query by date in collection with 5
shards. When i try same query on collection with only one shard everything
works.

*query:*

/solr/search_cz/select?q=*:*&group=true&group.field=odjezd

*part of schema.xml*



...



*collection create *

/solr/admin/collections\?action\=CREATE\&autoAddReplicas\=true\&collection.configName\=search_cz\&maxShardsPerNode\=5\&name\=search_cz\&numShards\=5\&replicationFactor\=1\&router.field\=routing_key\&
router.name\=compositeId\&rule\=shard:\*,replica:\<2,node:\*


*stacktrace:*

org.apache.solr.common.SolrException: Invalid Date String:'Wed Nov 21
12:45:00 UTC 2018'
at 
org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:247)
at 
org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:226)
at 
org.apache.solr.schema.DatePointField.toNativeType(DatePointField.java:113)
at 
org.apache.solr.schema.DatePointField.readableToIndexed(DatePointField.java:184)
at 
org.apache.solr.search.grouping.distributed.command.GroupConverter.fromMutable(GroupConverter.java:57)
at 
org.apache.solr.search.grouping.distributed.command.SearchGroupsFieldCommand.result(SearchGroupsFieldCommand.java:128)
at 
org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:55)
at 
org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:37)
at 
org.apache.solr.search.grouping.CommandHandler.processResult(CommandHandler.java:208)
at 
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchFirstPhase(QueryComponent.java:1282)
at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:360)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.Queu

Re: Results grouping performance with groups.ngroups=true

[Mikhail Khludnev]

I mean, you might probably count the same counts by json facet *instead*
slow grouping count, like
https://issues.apache.org/jira/browse/SOLR-7036?focusedCommentId=15601789&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15601789



On Sun, Aug 12, 2018 at 7:09 AM SayantiGmail  wrote:

> Hi Mikhail
>
> Even after using json facets latency seems to be high if
> group.ngroups=true.
>
> Regards,
> Sayan
>
> > On 12 Aug 2018, at 02:07, Mikhail Khludnev  wrote:
> >
> > As far as I remember, groups facets can be calculated with json.facets a
> > way faster.
> >
> >> On Sat, Aug 11, 2018 at 1:43 PM SayantiGmail 
> wrote:
> >>
> >> Hi,
> >>
> >> The time taken to group results when the resultset has ~ 200k items is
> >> very high.
> >>
> >> Is there a way to optimize the performance.
> >> The group count and facet count is required.
> >>
> >> Regards,
> >> Sayan
> >>
> >>
> >>
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Results grouping performance with groups.ngroups=true

[SayantiGmail]

Hi Mikhail

Even after using json facets latency seems to be high if group.ngroups=true.

Regards,
Sayan

> On 12 Aug 2018, at 02:07, Mikhail Khludnev  wrote:
> 
> As far as I remember, groups facets can be calculated with json.facets a
> way faster.
> 
>> On Sat, Aug 11, 2018 at 1:43 PM SayantiGmail  wrote:
>> 
>> Hi,
>> 
>> The time taken to group results when the resultset has ~ 200k items is
>> very high.
>> 
>> Is there a way to optimize the performance.
>> The group count and facet count is required.
>> 
>> Regards,
>> Sayan
>> 
>> 
>> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev

Re: Results grouping performance with groups.ngroups=true

[Mikhail Khludnev]

As far as I remember, groups facets can be calculated with json.facets a
way faster.

On Sat, Aug 11, 2018 at 1:43 PM SayantiGmail  wrote:

> Hi,
>
> The time taken to group results when the resultset has ~ 200k items is
> very high.
>
> Is there a way to optimize the performance.
> The group count and facet count is required.
>
> Regards,
> Sayan
>
>
>

-- 
Sincerely yours
Mikhail Khludnev

Results grouping performance with groups.ngroups=true

[SayantiGmail]

Hi,

The time taken to group results when the resultset has ~ 200k items is very 
high.

Is there a way to optimize the performance.
The group count and facet count is required.

Regards,
Sayan

Re: Caching Solr Grouping Results

[Yasufumi Mizoguchi]

Hi,

Have you already tried "group.cache.percent" parameter? It might improve
grouping performance.
Or if you try CollapsingQParser, you can use expand component to acquire
all values in groups, I think.
(
https://lucene.apache.org/solr/guide/6_6/collapse-and-expand-results.html#collapse-and-expand-results
)

thanks,
Yasufumi

2018年5月21日(月) 15:36 rubi.hali :

> Hi Yasufumi
>
> Thanks for the reply. Yes, you are correct. I also checked the code and it
> seems the same.
>
> We are facing performance issues due to grouping so wanted to be sure that
> we are not leaving out any possibility of caching the same in Query Result
> Cache.
>
> was just exploring field collapsing instead of grouping but It doesn't
> fulfill our requirement of having all values in a facet for a group instead
> of only grouped value. So we dont have any other choice but to go with
> grouping and may be write some custom cache provider :(
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Caching Solr Grouping Results

[rubi.hali]

Hi Yasufumi

Thanks for the reply. Yes, you are correct. I also checked the code and it
seems the same. 

We are facing performance issues due to grouping so wanted to be sure that
we are not leaving out any possibility of caching the same in Query Result
Cache.

was just exploring field collapsing instead of grouping but It doesn't
fulfill our requirement of having all values in a facet for a group instead
of only grouped value. So we dont have any other choice but to go with
grouping and may be write some custom cache provider :(





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Caching Solr Grouping Results

[Yasufumi Mizoguchi]

Hi,

I know few about groping component, but I think it is very hard. Because
query result cache has {query and conditions} -> {DocList} structure.
(
https://github.com/apache/lucene-solr/blob/e30264b31400a147507aabd121b1152020b8aa6d/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L120
)

But in caching grouping result, query result cache should have {query and
conditions} -> {grouped value, condition, etc...} -> {DocList} structure
cache, I think.

Thanks,
Yasufumi

2018年5月18日(金) 23:41 rubi.hali :

> Hi All
>
> Can somebody please explain if we can cache solr grouping results in query
> result cache as i dont see any inserts in query result cache once we
> enabled
> grouping?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Caching Solr Grouping Results

[rubi.hali]

Hi All

Can somebody please explain if we can cache solr grouping results in query
result cache as i dont see any inserts in query result cache once we enabled
grouping?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Also, would like to understand what are the ways to optimize for performance
at search time with LTR. Queries with terms (that fetch more results) lead
to very high latency with re-rank query even for reRankDocs=24. 

Is there best practices to reduce the latency?

Can fv cache help?


Should I increase the cache size?





-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli]

Thanks ilayaraja,

I updated the PR today integrating your and Alan's comments. Now it works
also in distributed mode. Please let me know what do you think :)

Cheers
Diego

On Wed, May 2, 2018, 17:46 ilayaraja  wrote:

> Figured out that offset is used as part of the grouping patch which I
> applied
> (SOLR-8776) :
> solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
> +  if (query instanceof AbstractReRankQuery){
> +topNGroups = cmd.getOffset() +
> ((AbstractReRankQuery)query).getReRankDocs();
> +  } else {
> +topNGroups = cmd.getOffset() + cmd.getLen();
>
>
>
>
>
>
> -
> --Ilay
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Figured out that offset is used as part of the grouping patch which I applied
(SOLR-8776) :
solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
+  if (query instanceof AbstractReRankQuery){
+topNGroups = cmd.getOffset() +
((AbstractReRankQuery)query).getReRankDocs();
+  } else {
+topNGroups = cmd.getOffset() + cmd.getLen();






-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

*
"Top K shouldn't start from the "start" parameter, if it does, it is a bug.
"***

1. I clearly see that LTR do re-rank based on the start parameter.
2. When reRankDocs=24, pageSize=24, I still get the second page of results
re-ranked by ltr plugin when I query with start=24.


Alessandro Benedetti wrote
> Are you using SolrCloud or any distributed search ?
> 
> If you are using just a single Solr instance, LTR should have no problem
> with pagination.
> The re-rank involves the top K and then you paginate.
> So if a document from the original score page 1 ends up in page 3, you
> will
> see it at page three.
> have you verified that : "Say, if an item (Y) from second page is moved to
> first page after 
> re-ranking, while an item (X) from first page is moved away from the first 
> page.  ?" 
> Top K shouldn't start from the "start" parameter, if it does, it is a bug.
> 
> The situation change a little with distributed search where you can
> experiment this behaviour : 
> 
> *Pagination*
> Let’s explore the scenario on a single Solr node and on a sharded
> architecture.
> 
> SINGLE SOLR NODE
> 
> reRankDocs=15
> rows=10
> This means each page is composed by 10 results.
> What happens when we hit the page 2 ?
> The first 5 documents in the search results will have been rescored and
> affected by the reranking.
> The latter 5 documents will preserve the original score and original
> ranking.
> 
> e.g.
> Doc 11 – score= 1.2
> Doc 12 – score= 1.1
> Doc 13 – score= 1.0
> Doc 14 – score= 0.9
> Doc 15 – score= 0.8
> Doc 16 – score= 5.7
> Doc 17 – score= 5.6
> Doc 18 – score= 5.5
> Doc 19 – score= 4.6
> Doc 20 – score= 2.4
> This means that score(15) could be < score(16), but document 15 and 16 are
> still in the expected order.
> The reason is that the top 15 documents are rescored and reranked and the
> rest is left unchanged.
> 
> *SHARDED ARCHITECTURE*
> 
> reRankDocs=15
> rows=10
> Shards number=2
> When looking for the page 2, Solr will trigger queries to she shards to
> collect 2 pages per shard :
> Shard1 : 10 ReRanked docs (page1) + 5 ReRanked docs + 5 OriginalScored
> docs
> (page2)
> Shard2 : 10 ReRanked docs (page1) + 5 ReRanked docs + 5 OriginalScored
> docs
> (page2)
> 
> The the results will be merged, and possibly, original scored search
> results
> can top up reranked docs.
> A possible solution could be to normalise the scores to prevent any
> possibility that a reranked result is surpassed by original scored ones.
> 
> Note: The problem is going to happen after you reach rows * page >
> reRankDocs. In situations when reRankDocs is quite high , the problem will
> occur only in deep paging.
> 
> 
> 
> -
> ---
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html





-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Alessandro Benedetti]

Are you using SolrCloud or any distributed search ?

If you are using just a single Solr instance, LTR should have no problem
with pagination.
The re-rank involves the top K and then you paginate.
So if a document from the original score page 1 ends up in page 3, you will
see it at page three.
have you verified that : "Say, if an item (Y) from second page is moved to
first page after 
re-ranking, while an item (X) from first page is moved away from the first 
page.  ?" 
Top K shouldn't start from the "start" parameter, if it does, it is a bug.

The situation change a little with distributed search where you can
experiment this behaviour : 

*Pagination*
Let’s explore the scenario on a single Solr node and on a sharded
architecture.

SINGLE SOLR NODE

reRankDocs=15
rows=10
This means each page is composed by 10 results.
What happens when we hit the page 2 ?
The first 5 documents in the search results will have been rescored and
affected by the reranking.
The latter 5 documents will preserve the original score and original
ranking.

e.g.
Doc 11 – score= 1.2
Doc 12 – score= 1.1
Doc 13 – score= 1.0
Doc 14 – score= 0.9
Doc 15 – score= 0.8
Doc 16 – score= 5.7
Doc 17 – score= 5.6
Doc 18 – score= 5.5
Doc 19 – score= 4.6
Doc 20 – score= 2.4
This means that score(15) could be < score(16), but document 15 and 16 are
still in the expected order.
The reason is that the top 15 documents are rescored and reranked and the
rest is left unchanged.

*SHARDED ARCHITECTURE*

reRankDocs=15
rows=10
Shards number=2
When looking for the page 2, Solr will trigger queries to she shards to
collect 2 pages per shard :
Shard1 : 10 ReRanked docs (page1) + 5 ReRanked docs + 5 OriginalScored docs
(page2)
Shard2 : 10 ReRanked docs (page1) + 5 ReRanked docs + 5 OriginalScored docs
(page2)

The the results will be merged, and possibly, original scored search results
can top up reranked docs.
A possible solution could be to normalise the scores to prevent any
possibility that a reranked result is surpassed by original scored ones.

Note: The problem is going to happen after you reach rows * page >
reRankDocs. In situations when reRankDocs is quite high , the problem will
occur only in deep paging.



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Between, I have applied the patch on top of solr 7.2.1 and it worked well for
me though the Test Cases were failing, yet to see why.

On another note, LTR with reRankDocs>page_size seems to create issue. For
example, Say my page_size=24 and reRankDocs=48. 

For first query with start=0, it returns 24 reranked results from top 2
result pages.
Say, if an item (Y) from second page is moved to first page after
re-ranking, while an item (X) from first page is moved away from the first
page. 

For second query with start=24, reRankDocs=48, it returns me second page of
results from results between second and third page that does not have item
X.

So eventually, I do not see item X from first page or next page of results.
Is n't it?

How do we solve this?



-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Between, I have applied the patch on top of solr 7.2.1 and it worked well for
me though the Test Cases were failing, yet to see why.

On another note, LTR with reRankDocs>page_size seems to create issue. For
example, Say my page_size=24 and reRankDocs=48. 

For first query with start=0, it returns 24 reranked results from top 2
result pages.
Say, if an item (Y) from second page is moved to first page after
re-ranking, while an item (X) from first page is moved away from the first
page. 

For second query with start=24, reRankDocs=48, it returns me second page of
results from results between second and third page that does not have item
X.

So eventually, I do not see item X from first page or next page of results.
Is n't it?

How do we solve this?



-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Nitin Kumar]

 Can anybody please share, ltr and group together works fine in which solr
version.

On Wed, Apr 18, 2018 at 3:47 PM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
dceccarel...@bloomberg.net> wrote:

> I just updated the PR to upstream - I still have to fix some things in
> distribute mode, but unit tests in non distribute mode works.
>
> Hope this helps,
> Diego
>
> From: solr-user@lucene.apache.org At: 04/15/18 03:37:54To:
> solr-user@lucene.apache.org
> Subject: Re: Learning to Rank (LTR) with grouping
>
> People sometimes fill in the Fix/Version field when they're creating
> the JIRA, since anyone can open a JIRA it's hard to control. I took
> that out just now.
>
> Basically if the "Resolution" field doesn't indicate it's fixed, you
> should assume that it hasn't been addressed.
>
> Patches welcome.
>
> Best,
> Erick
>
> On Tue, Apr 3, 2018 at 9:11 AM, ilayaraja  wrote:
> > Thanks Roopa.
> >
> > I was expecting that the issue has been fixed in solr 7.0 as per here
> > https://issues.apache.org/jira/browse/SOLR-8776.
> >
> > Let me see why it is still not working on solr-ltr-7.2.1
> >
> >
> >
> > -
> > --Ilay
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>
>

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

I just updated the PR to upstream - I still have to fix some things in 
distribute mode, but unit tests in non distribute mode works. 

Hope this helps, 
Diego 

From: solr-user@lucene.apache.org At: 04/15/18 03:37:54To:  
solr-user@lucene.apache.org
Subject: Re: Learning to Rank (LTR) with grouping

People sometimes fill in the Fix/Version field when they're creating
the JIRA, since anyone can open a JIRA it's hard to control. I took
that out just now.

Basically if the "Resolution" field doesn't indicate it's fixed, you
should assume that it hasn't been addressed.

Patches welcome.

Best,
Erick

On Tue, Apr 3, 2018 at 9:11 AM, ilayaraja  wrote:
> Thanks Roopa.
>
> I was expecting that the issue has been fixed in solr 7.0 as per here
> https://issues.apache.org/jira/browse/SOLR-8776.
>
> Let me see why it is still not working on solr-ltr-7.2.1
>
>
>
> -
> --Ilay
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Alessandro Benedetti]

Thanks for the response Shawn !

In relation to this : 
"I feel fairly sure that most of them are unwilling to document their
skills.  
If information like that is documented, it might saddle a committer with 
an obligation to work on issues affecting those areas when they may not 
have the free time available to cover that obligation. "

I understand your point.
I was referring to pure Lucene/Solr modules interest/expertise more than
skills but I get that "it might saddle a committer with 
an obligation to work on issues affecting those areas when they may not 
have the free time available to cover that obligation."

It shouldn't transmit an obligation ( as no contributor operates under any
SLA but purely passion driven ) but it might be a "suggestion" .
I was thinking to some way to avoid such long standing Jiras.
Let's pick this issue as an example.
>From the little of my opinion I believe it is quite useful.
The last activity is from 22/May/17 15:23 and no committer commented after
that.
I would assume that committers with interest or expertise on Learning To
Rank or Grouping initially didn't have free time to evaluate the patch and
then maybe they just forgot.
Having some sort of tagging based on expertise could at least avoid the
"forget" part ?
Or the contributor should viralize the issue and get as much "votes" from
the community as possible to validate an issue to be sexy ?
Just thinking loudly, it was just an idea ( and I am not completely sure it
could help) but I believe as a community we should manage a little bit
better contributions, of course I am open to any idea and perspective.

Cheers




-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Shawn Heisey]

On 4/17/2018 5:35 AM, Alessandro Benedetti wrote:

Apache Lucene/Solr is a big project, is there anywhere in the official
Apache Lucene/Solr website where each committer list the modules of
interest/expertise ?


No, there is no repository like that.  Each committer knows what their 
own expertise is of course, and sometimes may know a little bit about 
the expertise of a few others, but there is nothing documented.  I feel 
fairly sure that most of them are unwilling to document their skills.  
If information like that is documented, it might saddle a committer with 
an obligation to work on issues affecting those areas when they may not 
have the free time available to cover that obligation.



I understand that all of us contributors ( and committers) are just
volunteers, so no SLA is expected at all, but did the fact of the fixed
version already assigned affect the address of that Jira issue ?


The fix version is initially assigned by the person who opens the jira.  
When an issue is opened, that field should not be populated, but we 
can't expect everybody to know that.


If one of the committers happens to notice that there is a fix version 
but nobody is actually working on the issue, that may get cleared out.  
A committer will usually only enter one or more values in the fix 
version field if they are reasonably certain that they will actually get 
a fix committed to those specific releases.  For this reason, that field 
is often left blank until the change is actually ready.  Releases are 
not scheduled in advance, so until a release manager has volunteered and 
started work on a release, we never know when it's going to happen.


Thanks,
Shawn

Re: Learning to Rank (LTR) with grouping

[Alessandro Benedetti]

Hi Erick,
I have a curiosity/suggestion regarding how to speed up pending( or
forgotten ) Jiras,
is there a way to find out the most suitable committer(s) for the task and
tag them ?

Apache Lucene/Solr is a big project, is there anywhere in the official
Apache Lucene/Solr website where each committer list the modules of
interest/expertise ?
In this way when a contrbutor create a Jira and attach a patch, the
committers could get a notification if the module involving the Jira is one
of their interest.
This could be done manually ( the contributor check the committers interests
and manually tag them in the Jira) or automatically ( integrating Jira
modules with this "Interests list" in some way) .
Happy to help in this direction.

I understand that all of us contributors ( and committers) are just
volunteers, so no SLA is expected at all, but did the fact of the fixed
version already assigned affect the address of that Jira issue ?


Cheers
 



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Erick Erickson]

People sometimes fill in the Fix/Version field when they're creating
the JIRA, since anyone can open a JIRA it's hard to control. I took
that out just now.

Basically if the "Resolution" field doesn't indicate it's fixed, you
should assume that it hasn't been addressed.

Patches welcome.

Best,
Erick

On Tue, Apr 3, 2018 at 9:11 AM, ilayaraja  wrote:
> Thanks Roopa.
>
> I was expecting that the issue has been fixed in solr 7.0 as per here
> https://issues.apache.org/jira/browse/SOLR-8776.
>
> Let me see why it is still not working on solr-ltr-7.2.1
>
>
>
> -
> --Ilay
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re:Support LTR RankQuery with Grouping

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Patch has not been merged yet, it is available here: 

https://github.com/apache/lucene-solr/pull/162

You can try to apply the patch on the current master and see if it fixes.
Please let us know if you have any questions. 

Cheers,
Diego


From: solr-user@lucene.apache.org At: 04/05/18 05:36:20To:  
solr-user@lucene.apache.org
Subject: Support LTR RankQuery with Grouping

I am facing issue with LTR query not supported with grouping.

I see the patch for this has been raised here
https://issues.apache.org/jira/browse/SOLR-8776

Is it available in solr/master (7.2.2) now?

Looks like this patch is not merged yet. 


-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Support LTR RankQuery with Grouping

[ilayaraja]

I am facing issue with LTR query not supported with grouping.

I see the patch for this has been raised here
https://issues.apache.org/jira/browse/SOLR-8776

Is it available in solr/master (7.2.2) now?

Looks like this patch is not merged yet. 




-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Thanks Roopa.

I was expecting that the issue has been fixed in solr 7.0 as per here
https://issues.apache.org/jira/browse/SOLR-8776.

Let me see why it is still not working on solr-ltr-7.2.1



-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi Ilay,

I am still on Solr 6.6.0 and did not patch the grouping fix.
I implemented a temporary workaround solution to have 2 async request from
the web application 1st with grouping 2nd without grouping and merged the
results.
This solution worked for my case as we were getting grouping results for
specific tiles in the page.


Roopa


On Mon, Apr 2, 2018 at 2:57 AM, ilayaraja  wrote:

> Hi Roopa & Deigo,
>
>  I am facing same issue with grouping. Currently, am on Solr 7.2.1 but
> still
> see that grouping with LTR is not working. Did you apply it as patch or the
> latest solr version has the fix already?
>
> Ilay
>
>
>
> -
> --Ilay
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Learning to Rank (LTR) with grouping

[ilayaraja]

Hi Roopa & Deigo,

 I am facing same issue with grouping. Currently, am on Solr 7.2.1 but still
see that grouping with LTR is not working. Did you apply it as patch or the
latest solr version has the fix already?

Ilay



-
--Ilay
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Grouping on Exact Match

[Emir Arnautović]

Is it possible that this field used to be text_en field and later you changed 
it to string_lower but did not reindex. If you open schema explorer in your 
admin GUI, what tokens do you see for this field?

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 25 Jan 2018, at 12:25, Gopesh Sharma  wrote:
> 
> Hello Emir,
> 
> Service.Solr/partners/select/?wt=json&fq=-partnerProjectCount_num:0&bf=searchScore_num&sort=score+desc&q=Construction%20Company&group=true&group.field=partnerName_text_en&group.ngroups=true&group.limit=100&rows=50&start=0&indent=true
>  
> 
>  stored="true" multiValued="false" />
> 
> I am using the above query.
> 
> Thanks,
> Gopesh Sharma
> 
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com] 
> Sent: Thursday, January 25, 2018 4:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Grouping on Exact Match
> 
> Hi Gopesh,
> No it is not - at least not in a way I was thinking. I should have been more 
> precise - I meant tokenized. So you are grouping on the field that uses this  
> field type? Can you share query?
> 
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
> Elasticsearch Consulting Support Training - 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsematext.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7C04a138e3daf7423ba05f08d563e2e1ef%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636524748464316159&sdata=wuZfMQmGJbnHkPVMjtnUysxHud4U9wP57tDh9kPKNig%3D&reserved=0
> 
> 
> 
>> On 25 Jan 2018, at 11:57, Gopesh Sharma  wrote:
>> 
>> Hello Emir,
>> 
>> Thanks for the reply.
>> 
>> So if I am using below field type for the field, that means its analyzed?
>> 
>> > positionIncrementGap="100" autoGeneratePhraseQueries="true">
>>   
>>   
>>   
>>   
>>   
>>   
>>   
>>   
>> 
>> 
>> Thanks,
>> Gopesh Sharma
>> 
>> -Original Message-
>> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
>> Sent: Thursday, January 25, 2018 4:16 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Grouping on Exact Match
>> 
>> Hi Gopesh,
>> You are probably grouping on field that is analysed so “Consulting” is group 
>> term. What you need to do is to have name field that is not alalysed and 
>> group on that field. If you want to “group” on query input, that is not 
>> grouping - you simply use phrase query and all results are of that group.
>> 
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
>> Elasticsearch Consulting Support Training - 
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsemate
>> xt.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7Cda7c34701f32465
>> 883cc08d563e0ddd3%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C6365247
>> 39799609813&sdata=yXhlw36459RoXTKifDyGPVy1bgTpcVWOdHfViuGJgg8%3D&reser
>> ved=0
>> 
>> 
>> 
>>> On 25 Jan 2018, at 11:41, Gopesh Sharma  wrote:
>>> 
>>> Hello All,
>>> 
>>> I am grouping the results but the groups are not happening on the 
>>> exact match. For Example : I have 5 documents with name Construction 
>>> Company, Construction Tower, Tower Company, Tower House and again 
>>> Construction Company. If I search for Construction Company with 
>>> grouping I am getting result as
>>> 
>>> 
>>> *   Construction having 3 documents -  Construction Company, Construction 
>>> Tower and Construction Company with group value construction.
>>> 
>>> Is there any way I can get only two results with group value as exact match 
>>> of the search item.
>>> 
>>> Thanks,
>>> Gopesh Sharma
>> 
> 

RE: Grouping on Exact Match

[Gopesh Sharma]

Hello Emir,

Service.Solr/partners/select/?wt=json&fq=-partnerProjectCount_num:0&bf=searchScore_num&sort=score+desc&q=Construction%20Company&group=true&group.field=partnerName_text_en&group.ngroups=true&group.limit=100&rows=50&start=0&indent=true
 



I am using the above query.

Thanks,
Gopesh Sharma

-Original Message-
From: Emir Arnautović [mailto:emir.arnauto...@sematext.com] 
Sent: Thursday, January 25, 2018 4:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Grouping on Exact Match

Hi Gopesh,
No it is not - at least not in a way I was thinking. I should have been more 
precise - I meant tokenized. So you are grouping on the field that uses this  
field type? Can you share query?

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch 
Consulting Support Training - 
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsematext.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7C04a138e3daf7423ba05f08d563e2e1ef%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636524748464316159&sdata=wuZfMQmGJbnHkPVMjtnUysxHud4U9wP57tDh9kPKNig%3D&reserved=0



> On 25 Jan 2018, at 11:57, Gopesh Sharma  wrote:
> 
> Hello Emir,
> 
> Thanks for the reply.
> 
> So if I am using below field type for the field, that means its analyzed?
> 
>  positionIncrementGap="100" autoGeneratePhraseQueries="true">
>
>
>
>
>
>
>
>
> 
> 
> Thanks,
> Gopesh Sharma
> 
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
> Sent: Thursday, January 25, 2018 4:16 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Grouping on Exact Match
> 
> Hi Gopesh,
> You are probably grouping on field that is analysed so “Consulting” is group 
> term. What you need to do is to have name field that is not alalysed and 
> group on that field. If you want to “group” on query input, that is not 
> grouping - you simply use phrase query and all results are of that group.
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
> Elasticsearch Consulting Support Training - 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsemate
> xt.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7Cda7c34701f32465
> 883cc08d563e0ddd3%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C6365247
> 39799609813&sdata=yXhlw36459RoXTKifDyGPVy1bgTpcVWOdHfViuGJgg8%3D&reser
> ved=0
> 
> 
> 
>> On 25 Jan 2018, at 11:41, Gopesh Sharma  wrote:
>> 
>> Hello All,
>> 
>> I am grouping the results but the groups are not happening on the 
>> exact match. For Example : I have 5 documents with name Construction 
>> Company, Construction Tower, Tower Company, Tower House and again 
>> Construction Company. If I search for Construction Company with 
>> grouping I am getting result as
>> 
>> 
>> *   Construction having 3 documents -  Construction Company, Construction 
>> Tower and Construction Company with group value construction.
>> 
>> Is there any way I can get only two results with group value as exact match 
>> of the search item.
>> 
>> Thanks,
>> Gopesh Sharma
> 

Re: Grouping on Exact Match

[Emir Arnautović]

Hi Gopesh,
No it is not - at least not in a way I was thinking. I should have been more 
precise - I meant tokenized. So you are grouping on the field that uses this  
field type? Can you share query?

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 25 Jan 2018, at 11:57, Gopesh Sharma  wrote:
> 
> Hello Emir,
> 
> Thanks for the reply.
> 
> So if I am using below field type for the field, that means its analyzed?
> 
>  positionIncrementGap="100" autoGeneratePhraseQueries="true">
>
>
>
>
>
>
>
>
> 
> 
> Thanks,
> Gopesh Sharma
> 
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com] 
> Sent: Thursday, January 25, 2018 4:16 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Grouping on Exact Match
> 
> Hi Gopesh,
> You are probably grouping on field that is analysed so “Consulting” is group 
> term. What you need to do is to have name field that is not alalysed and 
> group on that field. If you want to “group” on query input, that is not 
> grouping - you simply use phrase query and all results are of that group.
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
> Elasticsearch Consulting Support Training - 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsematext.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7Cda7c34701f32465883cc08d563e0ddd3%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636524739799609813&sdata=yXhlw36459RoXTKifDyGPVy1bgTpcVWOdHfViuGJgg8%3D&reserved=0
> 
> 
> 
>> On 25 Jan 2018, at 11:41, Gopesh Sharma  wrote:
>> 
>> Hello All,
>> 
>> I am grouping the results but the groups are not happening on the 
>> exact match. For Example : I have 5 documents with name Construction 
>> Company, Construction Tower, Tower Company, Tower House and again 
>> Construction Company. If I search for Construction Company with 
>> grouping I am getting result as
>> 
>> 
>> *   Construction having 3 documents -  Construction Company, Construction 
>> Tower and Construction Company with group value construction.
>> 
>> Is there any way I can get only two results with group value as exact match 
>> of the search item.
>> 
>> Thanks,
>> Gopesh Sharma
> 

RE: Grouping on Exact Match

[Gopesh Sharma]

Hello Emir,

Thanks for the reply.

So if I am using below field type for the field, that means its analyzed?












Thanks,
Gopesh Sharma

-Original Message-
From: Emir Arnautović [mailto:emir.arnauto...@sematext.com] 
Sent: Thursday, January 25, 2018 4:16 PM
To: solr-user@lucene.apache.org
Subject: Re: Grouping on Exact Match

Hi Gopesh,
You are probably grouping on field that is analysed so “Consulting” is group 
term. What you need to do is to have name field that is not alalysed and group 
on that field. If you want to “group” on query input, that is not grouping - 
you simply use phrase query and all results are of that group.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch 
Consulting Support Training - 
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsematext.com%2F&data=02%7C01%7CGopesh_Sharma%40gensler.com%7Cda7c34701f32465883cc08d563e0ddd3%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636524739799609813&sdata=yXhlw36459RoXTKifDyGPVy1bgTpcVWOdHfViuGJgg8%3D&reserved=0



> On 25 Jan 2018, at 11:41, Gopesh Sharma  wrote:
> 
> Hello All,
> 
> I am grouping the results but the groups are not happening on the 
> exact match. For Example : I have 5 documents with name Construction 
> Company, Construction Tower, Tower Company, Tower House and again 
> Construction Company. If I search for Construction Company with 
> grouping I am getting result as
> 
> 
>  *   Construction having 3 documents -  Construction Company, Construction 
> Tower and Construction Company with group value construction.
> 
> Is there any way I can get only two results with group value as exact match 
> of the search item.
> 
> Thanks,
> Gopesh Sharma

Re: Grouping on Exact Match

[Emir Arnautović]

Hi Gopesh,
You are probably grouping on field that is analysed so “Consulting” is group 
term. What you need to do is to have name field that is not alalysed and group 
on that field. If you want to “group” on query input, that is not grouping - 
you simply use phrase query and all results are of that group.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 25 Jan 2018, at 11:41, Gopesh Sharma  wrote:
> 
> Hello All,
> 
> I am grouping the results but the groups are not happening on the exact 
> match. For Example : I have 5 documents with name Construction Company, 
> Construction Tower, Tower Company, Tower House and again Construction 
> Company. If I search for Construction Company with grouping I am getting 
> result as
> 
> 
>  *   Construction having 3 documents -  Construction Company, Construction 
> Tower and Construction Company with group value construction.
> 
> Is there any way I can get only two results with group value as exact match 
> of the search item.
> 
> Thanks,
> Gopesh Sharma

Grouping on Exact Match

[Gopesh Sharma]

Hello All,

I am grouping the results but the groups are not happening on the exact match. 
For Example : I have 5 documents with name Construction Company, Construction 
Tower, Tower Company, Tower House and again Construction Company. If I search 
for Construction Company with grouping I am getting result as


  *   Construction having 3 documents -  Construction Company, Construction 
Tower and Construction Company with group value construction.

Is there any way I can get only two results with group value as exact match of 
the search item.

Thanks,
Gopesh Sharma

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi Diego,

I tried collapsing, unfortunately we are using a third party tool for Query
processing, that does not support collapsing. Hence I am unable to go that
route.

Thank you,
Roopa

On Thu, Jan 11, 2018 at 2:59 PM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
dceccarel...@bloomberg.net> wrote:

> Roopa, did you try collapsing instead of grouping? it would work with ltr..
>
> From: solr-user@lucene.apache.org At: 01/11/18 16:48:26To:
> solr-user@lucene.apache.org
> Subject: Re: Learning to Rank (LTR) with grouping
>
> Solution that I implemented currently is:
> Since we have a web application which takes the solr results and display in
> the UI and I need LTR enabled for only one of the group,
> I am executing two parallel queries to Solr from web app.
> 1st query Get grouped results without LTR
> 2nd query Get results with no grouping + LTR
> Merge 1) & 2) in web app.
> Doing a performance test now. Since there are only 10 results for these
> queries and the queries are executed in parallel, I don't foresee any
> problems.
>
> Thanks!
> Roopa
>
>
> On Thu, Jan 4, 2018 at 10:24 AM, Roopa Rao  wrote:
>
> > Hi,
> >
> > Any guidance on this would be helpful.
> >
> > Thank you,
> > Roopa
> >
> > On Tue, Dec 19, 2017 at 8:47 PM, Roopa Rao  wrote:
> >
> >> Hi Diego,
> >>
> >> Thank you for looking into it further.
> >> We recently ported over to 6.6 version solely to use LTR feature as it
> is
> >> critical for us.
> >>
> >> Since its not working with grouping in the base version, I am trying to
> >> evaluate if there is any alternative way to make it work in 6.6 versus
> >> upgrading to 7.0.
> >>
> >> Any guidance you could provide on what can be done to use 6.6 with
> >> grouping + LTR or any alternatives would be helpful. Do I read your
> >> response as needing to go to 7.0 when you say upstream?
> >>
> >> Thank you,
> >> Roopa
> >>
> >>
> >> On Tue, Dec 19, 2017 at 1:37 PM, Diego Ceccarelli <
> >> diego.ceccare...@gmail.com> wrote:
> >>
> >>> Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
> >>> soon I'll update to upstream. Sorry about that.
> >>>
> >>> On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
> >>> > Hi -
> >>> >
> >>> > I merged the code from the bloomberg master-solr-8776 branch to
> >>> branch_6_6
> >>> > on Solr.
> >>> >
> >>> > When I tried to compile the solr source code, I am getting multiple
> >>> > compilation errors (Attached), which seems to be due to the fact that
> >>> the
> >>> > branch master-solr-8776 may not be compatible with branch_6_6.
> >>> >
> >>> > Could you please provide your input if master-solr-8776 is compatible
> >>> with
> >>> > branch_6_6?
> >>> >
> >>> > If this is not the case then how to proceed with using fix in
> >>> > master-solr-8776 with branch_6_6 can a new patch be created for this?
> >>> >
> >>> > Thank you,
> >>> > Roopa
> >>> >
> >>> > On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao 
> wrote:
> >>> >>
> >>> >> Hi Diego,
> >>> >>
> >>> >> Thank you,
> >>> >>
> >>> >> I am interested in reranking the documents inside one of the groups.
> >>> >>
> >>> >> I will try the options you mentioned here.
> >>> >>
> >>> >> Thank you,
> >>> >> Roopa
> >>> >>
> >>> >> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/
> LONDON)
> >>> >>  wrote:
> >>> >>>
> >>> >>> Hi Roopa,
> >>> >>>
> >>> >>> If you look at the diff:
> >>> >>>
> >>> >>> https://github.com/apache/lucene-solr/pull/162/files
> >>> >>>
> >>> >>> I didn't change much in SolrIndexSearcher, you can try to skip the
> >>> file
> >>> >>> when applying the patch and redo the changes after.
> >>> >>>
> >>> >>> Alternatively, the feature branch is available here:
> >>> >>>
> >>> >>> https://github.com/bloomberg/lucene-solr/commits/master-solr-

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Roopa, did you try collapsing instead of grouping? it would work with ltr.. 

From: solr-user@lucene.apache.org At: 01/11/18 16:48:26To:  
solr-user@lucene.apache.org
Subject: Re: Learning to Rank (LTR) with grouping

Solution that I implemented currently is:
Since we have a web application which takes the solr results and display in
the UI and I need LTR enabled for only one of the group,
I am executing two parallel queries to Solr from web app.
1st query Get grouped results without LTR
2nd query Get results with no grouping + LTR
Merge 1) & 2) in web app.
Doing a performance test now. Since there are only 10 results for these
queries and the queries are executed in parallel, I don't foresee any
problems.

Thanks!
Roopa


On Thu, Jan 4, 2018 at 10:24 AM, Roopa Rao  wrote:

> Hi,
>
> Any guidance on this would be helpful.
>
> Thank you,
> Roopa
>
> On Tue, Dec 19, 2017 at 8:47 PM, Roopa Rao  wrote:
>
>> Hi Diego,
>>
>> Thank you for looking into it further.
>> We recently ported over to 6.6 version solely to use LTR feature as it is
>> critical for us.
>>
>> Since its not working with grouping in the base version, I am trying to
>> evaluate if there is any alternative way to make it work in 6.6 versus
>> upgrading to 7.0.
>>
>> Any guidance you could provide on what can be done to use 6.6 with
>> grouping + LTR or any alternatives would be helpful. Do I read your
>> response as needing to go to 7.0 when you say upstream?
>>
>> Thank you,
>> Roopa
>>
>>
>> On Tue, Dec 19, 2017 at 1:37 PM, Diego Ceccarelli <
>> diego.ceccare...@gmail.com> wrote:
>>
>>> Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
>>> soon I'll update to upstream. Sorry about that.
>>>
>>> On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
>>> > Hi -
>>> >
>>> > I merged the code from the bloomberg master-solr-8776 branch to
>>> branch_6_6
>>> > on Solr.
>>> >
>>> > When I tried to compile the solr source code, I am getting multiple
>>> > compilation errors (Attached), which seems to be due to the fact that
>>> the
>>> > branch master-solr-8776 may not be compatible with branch_6_6.
>>> >
>>> > Could you please provide your input if master-solr-8776 is compatible
>>> with
>>> > branch_6_6?
>>> >
>>> > If this is not the case then how to proceed with using fix in
>>> > master-solr-8776 with branch_6_6 can a new patch be created for this?
>>> >
>>> > Thank you,
>>> > Roopa
>>> >
>>> > On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:
>>> >>
>>> >> Hi Diego,
>>> >>
>>> >> Thank you,
>>> >>
>>> >> I am interested in reranking the documents inside one of the groups.
>>> >>
>>> >> I will try the options you mentioned here.
>>> >>
>>> >> Thank you,
>>> >> Roopa
>>> >>
>>> >> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON)
>>> >>  wrote:
>>> >>>
>>> >>> Hi Roopa,
>>> >>>
>>> >>> If you look at the diff:
>>> >>>
>>> >>> https://github.com/apache/lucene-solr/pull/162/files
>>> >>>
>>> >>> I didn't change much in SolrIndexSearcher, you can try to skip the
>>> file
>>> >>> when applying the patch and redo the changes after.
>>> >>>
>>> >>> Alternatively, the feature branch is available here:
>>> >>>
>>> >>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>>> >>>
>>> >>> you could try to merge with that or cheery-pick my changes.
>>> >>>
>>> >>> Are you interested in reranking the groups or also in reranking the
>>> >>> documents inside each group?
>>> >>>
>>> >>> Cheers,
>>> >>> Diego
>>> >>>
>>> >>>
>>> >>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
>>> >>> solr-user@lucene.apache.org
>>> >>> Subject: Re: Learning to Rank (LTR) with grouping
>>> >>>
>>> >>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
>>> >>> critical.
>>> >>>
>>>

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Solution that I implemented currently is:
Since we have a web application which takes the solr results and display in
the UI and I need LTR enabled for only one of the group,
I am executing two parallel queries to Solr from web app.
1st query Get grouped results without LTR
2nd query Get results with no grouping + LTR
Merge 1) & 2) in web app.
Doing a performance test now. Since there are only 10 results for these
queries and the queries are executed in parallel, I don't foresee any
problems.

Thanks!
Roopa


On Thu, Jan 4, 2018 at 10:24 AM, Roopa Rao  wrote:

> Hi,
>
> Any guidance on this would be helpful.
>
> Thank you,
> Roopa
>
> On Tue, Dec 19, 2017 at 8:47 PM, Roopa Rao  wrote:
>
>> Hi Diego,
>>
>> Thank you for looking into it further.
>> We recently ported over to 6.6 version solely to use LTR feature as it is
>> critical for us.
>>
>> Since its not working with grouping in the base version, I am trying to
>> evaluate if there is any alternative way to make it work in 6.6 versus
>> upgrading to 7.0.
>>
>> Any guidance you could provide on what can be done to use 6.6 with
>> grouping + LTR or any alternatives would be helpful. Do I read your
>> response as needing to go to 7.0 when you say upstream?
>>
>> Thank you,
>> Roopa
>>
>>
>> On Tue, Dec 19, 2017 at 1:37 PM, Diego Ceccarelli <
>> diego.ceccare...@gmail.com> wrote:
>>
>>> Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
>>> soon I'll update to upstream. Sorry about that.
>>>
>>> On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
>>> > Hi -
>>> >
>>> > I merged the code from the bloomberg master-solr-8776 branch to
>>> branch_6_6
>>> > on Solr.
>>> >
>>> > When I tried to compile the solr source code, I am getting multiple
>>> > compilation errors (Attached), which seems to be due to the fact that
>>> the
>>> > branch master-solr-8776 may not be compatible with branch_6_6.
>>> >
>>> > Could you please provide your input if master-solr-8776 is compatible
>>> with
>>> > branch_6_6?
>>> >
>>> > If this is not the case then how to proceed with using fix in
>>> > master-solr-8776 with branch_6_6 can a new patch be created for this?
>>> >
>>> > Thank you,
>>> > Roopa
>>> >
>>> > On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:
>>> >>
>>> >> Hi Diego,
>>> >>
>>> >> Thank you,
>>> >>
>>> >> I am interested in reranking the documents inside one of the groups.
>>> >>
>>> >> I will try the options you mentioned here.
>>> >>
>>> >> Thank you,
>>> >> Roopa
>>> >>
>>> >> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON)
>>> >>  wrote:
>>> >>>
>>> >>> Hi Roopa,
>>> >>>
>>> >>> If you look at the diff:
>>> >>>
>>> >>> https://github.com/apache/lucene-solr/pull/162/files
>>> >>>
>>> >>> I didn't change much in SolrIndexSearcher, you can try to skip the
>>> file
>>> >>> when applying the patch and redo the changes after.
>>> >>>
>>> >>> Alternatively, the feature branch is available here:
>>> >>>
>>> >>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>>> >>>
>>> >>> you could try to merge with that or cheery-pick my changes.
>>> >>>
>>> >>> Are you interested in reranking the groups or also in reranking the
>>> >>> documents inside each group?
>>> >>>
>>> >>> Cheers,
>>> >>> Diego
>>> >>>
>>> >>>
>>> >>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
>>> >>> solr-user@lucene.apache.org
>>> >>> Subject: Re: Learning to Rank (LTR) with grouping
>>> >>>
>>> >>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
>>> >>> critical.
>>> >>>
>>> >>> Here are the steps I took on my mac:
>>> >>>
>>> >>> On branch branch_6_5
>>> >>>
>>> >>> Your branch is up-to-date with &#

Re: grouping in solr cloud shard replicas

[Shawn Heisey]

On 1/6/2018 11:25 AM, SANJAY. wrote:

Please let me know how to achieve group by in solr could env.
We tried grouping in solr cloud shard replicas to fetch unique search result
from solr for custom field we .

we are getting exception saying unexpected docvalues type "SORTED_SET
(expected SORTED)"


This typically happens when you index some data and then change the 
multivalued parameter on a field that has docValues without deleting the 
index and starting over.


When making that kind of change to the schema, you must completely 
delete the index directory for all cores in the collection, then reload 
the collection or restart Solr, and reindex from scratch.


This rather extreme step is required because the Lucene index records 
certain kinds of information about the docValues for each field that has 
them, and if what is expected doesn't match what's actually in the 
index, a severe error is encountered.


Thanks,
Shawn

grouping in solr cloud shard replicas

[SANJAY.]

Hi,

Please let me know how to achieve group by in solr could env.
We tried grouping in solr cloud shard replicas to fetch unique search result
from solr for custom field we .

we are getting exception saying unexpected docvalues type "SORTED_SET
(expected SORTED)"

We are using solr cloud and collection having 2 shared replicas .We have
created custom field type which is using solr.TextField class .

Please suggest me the best possible way to fetch the unique search result. 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi,

Any guidance on this would be helpful.

Thank you,
Roopa

On Tue, Dec 19, 2017 at 8:47 PM, Roopa Rao  wrote:

> Hi Diego,
>
> Thank you for looking into it further.
> We recently ported over to 6.6 version solely to use LTR feature as it is
> critical for us.
>
> Since its not working with grouping in the base version, I am trying to
> evaluate if there is any alternative way to make it work in 6.6 versus
> upgrading to 7.0.
>
> Any guidance you could provide on what can be done to use 6.6 with
> grouping + LTR or any alternatives would be helpful. Do I read your
> response as needing to go to 7.0 when you say upstream?
>
> Thank you,
> Roopa
>
>
> On Tue, Dec 19, 2017 at 1:37 PM, Diego Ceccarelli <
> diego.ceccare...@gmail.com> wrote:
>
>> Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
>> soon I'll update to upstream. Sorry about that.
>>
>> On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
>> > Hi -
>> >
>> > I merged the code from the bloomberg master-solr-8776 branch to
>> branch_6_6
>> > on Solr.
>> >
>> > When I tried to compile the solr source code, I am getting multiple
>> > compilation errors (Attached), which seems to be due to the fact that
>> the
>> > branch master-solr-8776 may not be compatible with branch_6_6.
>> >
>> > Could you please provide your input if master-solr-8776 is compatible
>> with
>> > branch_6_6?
>> >
>> > If this is not the case then how to proceed with using fix in
>> > master-solr-8776 with branch_6_6 can a new patch be created for this?
>> >
>> > Thank you,
>> > Roopa
>> >
>> > On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:
>> >>
>> >> Hi Diego,
>> >>
>> >> Thank you,
>> >>
>> >> I am interested in reranking the documents inside one of the groups.
>> >>
>> >> I will try the options you mentioned here.
>> >>
>> >> Thank you,
>> >> Roopa
>> >>
>> >> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON)
>> >>  wrote:
>> >>>
>> >>> Hi Roopa,
>> >>>
>> >>> If you look at the diff:
>> >>>
>> >>> https://github.com/apache/lucene-solr/pull/162/files
>> >>>
>> >>> I didn't change much in SolrIndexSearcher, you can try to skip the
>> file
>> >>> when applying the patch and redo the changes after.
>> >>>
>> >>> Alternatively, the feature branch is available here:
>> >>>
>> >>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>> >>>
>> >>> you could try to merge with that or cheery-pick my changes.
>> >>>
>> >>> Are you interested in reranking the groups or also in reranking the
>> >>> documents inside each group?
>> >>>
>> >>> Cheers,
>> >>> Diego
>> >>>
>> >>>
>> >>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
>> >>> solr-user@lucene.apache.org
>> >>> Subject: Re: Learning to Rank (LTR) with grouping
>> >>>
>> >>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
>> >>> critical.
>> >>>
>> >>> Here are the steps I took on my mac:
>> >>>
>> >>> On branch branch_6_5
>> >>>
>> >>> Your branch is up-to-date with 'origin/branch_6_5'
>> >>>
>> >>> patch -p1 -i 162.patch --dry-run
>> >>>
>> >>>
>> >>> I am getting Failures for certain Hunks
>> >>>
>> >>> Example:
>> >>>
>> >>> patching file
>> >>> solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
>> >>>
>> >>> Hunk #1 FAILED at 1471.
>> >>>
>> >>>
>> >>> Could you please give your input on how to apply this ticket as a
>> patch
>> >>> for
>> >>> branch_6_5 ?
>> >>>
>> >>>
>> >>> Thank you,
>> >>>
>> >>> Roopa
>> >>>
>> >>> On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:
>> >>>
>> >>> > Hi Diego,
>> >>> >
>> >>> > Thank you, I will look into this and see how I could patch this.
>> >>> >
>> >>> > Thank you for your quick response,
>> >>> > Roopa
>> >>> >
>> >>> >
>> >>> > On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
>> >>> > diego.ceccare...@gmail.com> wrote:
>> >>> >
>> >>> >> Hi Roopa,
>> >>> >>
>> >>> >> LTR is implemented using RankQuery, and at the moment grouping
>> doens't
>> >>> >> support RankQuery.
>> >>> >> I opened a jira item time ago
>> >>> >> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be
>> happy
>> >>> >> to receive feedback on that.  You can find the code here
>> >>> >> https://github.com/apache/lucene-solr/pull/162.
>> >>> >>
>> >>> >> Cheers,
>> >>> >> diego
>> >>> >>
>> >>> >> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao 
>> wrote:
>> >>> >> > Hi,
>> >>> >> >
>> >>> >> > I am using grouping and LTR together and the results are not
>> getting
>> >>> >> > re-rank as it does without grouping.
>> >>> >> >
>> >>> >> > I am passing &rq parameter.
>> >>> >> >
>> >>> >> > Does LTR work with grouping on?
>> >>> >> > Solr version 6.5
>> >>> >> >
>> >>> >> > Thank you,
>> >>> >> > Roopa
>> >>> >>
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>
>> >
>>
>
>

SOLR 6.5.1: timeAllowed parameter with grouping

[SOLR4189]

A month ago we upgraded our SOLR from 4.10.1 to 6.5.1. Now we want to use
timeAllowed parameter that was fixed in Solr 5. We check this parameter in
test servers and we don't understand if it works with group=true or not. 

* If we set group=false and timeAllowed=1 and query with too many terms:
sometimes it stops query, returns partial results and writes to log was
timeout and sometimes does nothing (oblivious to timeAllowed)

* If we set group=true and timeAllowed=1 and query with too many terms:
sometimes writes to log was timeout without stopping query and without
returning partial result  and sometimes does nothing (oblivious to
timeAllowed)

1) Can somebody explain this behavior? 
2) Does timeAllowed parameter work with grouping in SOLR-6.5.1 (We saw
SOLR-6156 issue that was updated 22/Apr/15 09:20 in the last time)?
3) If query was stopped, partial results will be saved in cache?




 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SOLR 6.5.1: timeAllowed parameter with grouping

[SOLR4189]

A month ago we upgraded our SOLR from 4.10.1 to 6.5.1. Now we want to use
timeAllowed parameter that was fixed in Solr 5. We check this parameter in
test servers and we don't understand if it works with group=true or not. 

* If we set group=false and timeAllowed=1 and query with too many terms:
sometimes it stops query, returns partial results and writes to log was
timeout and sometimes does nothing (oblivious to timeAllowed)

* If we set group=true and timeAllowed=1 and query with too many terms:
sometimes writes to log was timeout without stopping query and without
returning partial result  and sometimes does nothing (oblivious to
timeAllowed)

1) Can somebody explain this behavior? 
2) Does timeAllowed parameter work with grouping in SOLR-6.5.1 (We saw
SOLR-6156 issue that was updated 22/Apr/15 09:20 in the last time)?
3) If query was stopped, partial results will be saved in cache?




 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi Diego,

Thank you for looking into it further.
We recently ported over to 6.6 version solely to use LTR feature as it is
critical for us.

Since its not working with grouping in the base version, I am trying to
evaluate if there is any alternative way to make it work in 6.6 versus
upgrading to 7.0.

Any guidance you could provide on what can be done to use 6.6 with grouping
+ LTR or any alternatives would be helpful. Do I read your response as
needing to go to 7.0 when you say upstream?

Thank you,
Roopa


On Tue, Dec 19, 2017 at 1:37 PM, Diego Ceccarelli <
diego.ceccare...@gmail.com> wrote:

> Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
> soon I'll update to upstream. Sorry about that.
>
> On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
> > Hi -
> >
> > I merged the code from the bloomberg master-solr-8776 branch to
> branch_6_6
> > on Solr.
> >
> > When I tried to compile the solr source code, I am getting multiple
> > compilation errors (Attached), which seems to be due to the fact that the
> > branch master-solr-8776 may not be compatible with branch_6_6.
> >
> > Could you please provide your input if master-solr-8776 is compatible
> with
> > branch_6_6?
> >
> > If this is not the case then how to proceed with using fix in
> > master-solr-8776 with branch_6_6 can a new patch be created for this?
> >
> > Thank you,
> > Roopa
> >
> > On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:
> >>
> >> Hi Diego,
> >>
> >> Thank you,
> >>
> >> I am interested in reranking the documents inside one of the groups.
> >>
> >> I will try the options you mentioned here.
> >>
> >> Thank you,
> >> Roopa
> >>
> >> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON)
> >>  wrote:
> >>>
> >>> Hi Roopa,
> >>>
> >>> If you look at the diff:
> >>>
> >>> https://github.com/apache/lucene-solr/pull/162/files
> >>>
> >>> I didn't change much in SolrIndexSearcher, you can try to skip the file
> >>> when applying the patch and redo the changes after.
> >>>
> >>> Alternatively, the feature branch is available here:
> >>>
> >>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
> >>>
> >>> you could try to merge with that or cheery-pick my changes.
> >>>
> >>> Are you interested in reranking the groups or also in reranking the
> >>> documents inside each group?
> >>>
> >>> Cheers,
> >>> Diego
> >>>
> >>>
> >>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
> >>> solr-user@lucene.apache.org
> >>> Subject: Re: Learning to Rank (LTR) with grouping
> >>>
> >>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
> >>> critical.
> >>>
> >>> Here are the steps I took on my mac:
> >>>
> >>> On branch branch_6_5
> >>>
> >>> Your branch is up-to-date with 'origin/branch_6_5'
> >>>
> >>> patch -p1 -i 162.patch --dry-run
> >>>
> >>>
> >>> I am getting Failures for certain Hunks
> >>>
> >>> Example:
> >>>
> >>> patching file
> >>> solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
> >>>
> >>> Hunk #1 FAILED at 1471.
> >>>
> >>>
> >>> Could you please give your input on how to apply this ticket as a patch
> >>> for
> >>> branch_6_5 ?
> >>>
> >>>
> >>> Thank you,
> >>>
> >>> Roopa
> >>>
> >>> On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:
> >>>
> >>> > Hi Diego,
> >>> >
> >>> > Thank you, I will look into this and see how I could patch this.
> >>> >
> >>> > Thank you for your quick response,
> >>> > Roopa
> >>> >
> >>> >
> >>> > On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
> >>> > diego.ceccare...@gmail.com> wrote:
> >>> >
> >>> >> Hi Roopa,
> >>> >>
> >>> >> LTR is implemented using RankQuery, and at the moment grouping
> doens't
> >>> >> support RankQuery.
> >>> >> I opened a jira item time ago
> >>> >> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be
> happy
> >>> >> to receive feedback on that.  You can find the code here
> >>> >> https://github.com/apache/lucene-solr/pull/162.
> >>> >>
> >>> >> Cheers,
> >>> >> diego
> >>> >>
> >>> >> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao 
> wrote:
> >>> >> > Hi,
> >>> >> >
> >>> >> > I am using grouping and LTR together and the results are not
> getting
> >>> >> > re-rank as it does without grouping.
> >>> >> >
> >>> >> > I am passing &rq parameter.
> >>> >> >
> >>> >> > Does LTR work with grouping on?
> >>> >> > Solr version 6.5
> >>> >> >
> >>> >> > Thank you,
> >>> >> > Roopa
> >>> >>
> >>> >
> >>> >
> >>>
> >>>
> >>
> >
>

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli]

Hi Roopa, unfortunately I can't port the patch to the branch_6_6, but
soon I'll update to upstream. Sorry about that.

On Mon, Dec 18, 2017 at 7:52 PM, Roopa Rao  wrote:
> Hi -
>
> I merged the code from the bloomberg master-solr-8776 branch to branch_6_6
> on Solr.
>
> When I tried to compile the solr source code, I am getting multiple
> compilation errors (Attached), which seems to be due to the fact that the
> branch master-solr-8776 may not be compatible with branch_6_6.
>
> Could you please provide your input if master-solr-8776 is compatible with
> branch_6_6?
>
> If this is not the case then how to proceed with using fix in
> master-solr-8776 with branch_6_6 can a new patch be created for this?
>
> Thank you,
> Roopa
>
> On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:
>>
>> Hi Diego,
>>
>> Thank you,
>>
>> I am interested in reranking the documents inside one of the groups.
>>
>> I will try the options you mentioned here.
>>
>> Thank you,
>> Roopa
>>
>> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON)
>>  wrote:
>>>
>>> Hi Roopa,
>>>
>>> If you look at the diff:
>>>
>>> https://github.com/apache/lucene-solr/pull/162/files
>>>
>>> I didn't change much in SolrIndexSearcher, you can try to skip the file
>>> when applying the patch and redo the changes after.
>>>
>>> Alternatively, the feature branch is available here:
>>>
>>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>>>
>>> you could try to merge with that or cheery-pick my changes.
>>>
>>> Are you interested in reranking the groups or also in reranking the
>>> documents inside each group?
>>>
>>> Cheers,
>>> Diego
>>>
>>>
>>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
>>> solr-user@lucene.apache.org
>>> Subject: Re: Learning to Rank (LTR) with grouping
>>>
>>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
>>> critical.
>>>
>>> Here are the steps I took on my mac:
>>>
>>> On branch branch_6_5
>>>
>>> Your branch is up-to-date with 'origin/branch_6_5'
>>>
>>> patch -p1 -i 162.patch --dry-run
>>>
>>>
>>> I am getting Failures for certain Hunks
>>>
>>> Example:
>>>
>>> patching file
>>> solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
>>>
>>> Hunk #1 FAILED at 1471.
>>>
>>>
>>> Could you please give your input on how to apply this ticket as a patch
>>> for
>>> branch_6_5 ?
>>>
>>>
>>> Thank you,
>>>
>>> Roopa
>>>
>>> On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:
>>>
>>> > Hi Diego,
>>> >
>>> > Thank you, I will look into this and see how I could patch this.
>>> >
>>> > Thank you for your quick response,
>>> > Roopa
>>> >
>>> >
>>> > On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
>>> > diego.ceccare...@gmail.com> wrote:
>>> >
>>> >> Hi Roopa,
>>> >>
>>> >> LTR is implemented using RankQuery, and at the moment grouping doens't
>>> >> support RankQuery.
>>> >> I opened a jira item time ago
>>> >> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
>>> >> to receive feedback on that.  You can find the code here
>>> >> https://github.com/apache/lucene-solr/pull/162.
>>> >>
>>> >> Cheers,
>>> >> diego
>>> >>
>>> >> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
>>> >> > Hi,
>>> >> >
>>> >> > I am using grouping and LTR together and the results are not getting
>>> >> > re-rank as it does without grouping.
>>> >> >
>>> >> > I am passing &rq parameter.
>>> >> >
>>> >> > Does LTR work with grouping on?
>>> >> > Solr version 6.5
>>> >> >
>>> >> > Thank you,
>>> >> > Roopa
>>> >>
>>> >
>>> >
>>>
>>>
>>
>

Re: using rank queries(rq) with grouping in solr cloud

[Diego Ceccarelli]

Hi Tomerg,

1. Did you consider using the collapse component?
https://lucene.apache.org/solr/guide/6_6/collapse-and-expand-results.html
it is compatible with rq.

2. If you implement group reranking as a separate component you will
end up with a lot of code duplicated from QueryComponent, you could
use SOLR-8776 - I'm going to update it to master soon.

Another possible solution is to have a component that asks for the top
$rerank groups to the shards and then just do the reranking on top of
them in the federator, but it could be expensive.

Cheers,
Diego



On Fri, Dec 15, 2017 at 9:46 PM, tomerg  wrote:
> hey,
>
> i'm using solr 6.5.1 with solrCloud mode.
> i use grouping for my results.
> i want to use rank query(rq) in order to rerank the top groups(with ltr).
> it's ok for me to rerank the groups  only by reranking one of the documents
> in the group.
> i saw in issue SOLR-8776 that  rank queries doesn't  support  grouping.
> (link here: https://issues.apache.org/jira/browse/SOLR-8776).
>
> so i have a few questions:
> 1. there is some way to bypass this problem(or use some other existing
> features of solr to achieve similar results?
> 2. if there is no other way, i would like  to implement a component to
> achieve this functionality(i don't want to patch the code of solr itself).
> do you have a suggestion what might be the best way to implement a rerank of
> groups in cloud mode?
> can i implement something that rerank the groups for every shard before
> merging  or there is a way to create component that rerank only the merged
> result list from the shards?
>
> thanks,
> tomerg
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi -

I merged the code from the bloomberg master-solr-8776 branch to branch_6_6
on Solr.

When I tried to compile the solr source code, I am getting multiple
compilation errors (Attached), which seems to be due to the fact that the
branch master-solr-8776 may not be compatible with branch_6_6.

Could you please provide your input if master-solr-8776 is compatible with
branch_6_6?

If this is not the case then how to proceed with using fix in
master-solr-8776 with branch_6_6 can a new patch be created for this?

Thank you,
Roopa

On Mon, Dec 11, 2017 at 9:54 AM, Roopa Rao  wrote:

> Hi Diego,
>
> Thank you,
>
> I am interested in reranking the documents inside one of the groups.
>
> I will try the options you mentioned here.
>
> Thank you,
> Roopa
>
> On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
> dceccarel...@bloomberg.net> wrote:
>
>> Hi Roopa,
>>
>> If you look at the diff:
>>
>> https://github.com/apache/lucene-solr/pull/162/files
>>
>> I didn't change much in SolrIndexSearcher, you can try to skip the file
>> when applying the patch and redo the changes after.
>>
>> Alternatively, the feature branch is available here:
>>
>> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>>
>> you could try to merge with that or cheery-pick my changes.
>>
>> Are you interested in reranking the groups or also in reranking the
>> documents inside each group?
>>
>> Cheers,
>> Diego
>>
>>
>> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
>> solr-user@lucene.apache.org
>> Subject: Re: Learning to Rank (LTR) with grouping
>>
>> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
>> critical.
>>
>> Here are the steps I took on my mac:
>>
>> On branch branch_6_5
>>
>> Your branch is up-to-date with 'origin/branch_6_5'
>>
>> patch -p1 -i 162.patch --dry-run
>>
>>
>> I am getting Failures for certain Hunks
>>
>> Example:
>>
>> patching file
>> solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
>>
>> Hunk #1 FAILED at 1471.
>>
>>
>> Could you please give your input on how to apply this ticket as a patch
>> for
>> branch_6_5 ?
>>
>>
>> Thank you,
>>
>> Roopa
>>
>> On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:
>>
>> > Hi Diego,
>> >
>> > Thank you, I will look into this and see how I could patch this.
>> >
>> > Thank you for your quick response,
>> > Roopa
>> >
>> >
>> > On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
>> > diego.ceccare...@gmail.com> wrote:
>> >
>> >> Hi Roopa,
>> >>
>> >> LTR is implemented using RankQuery, and at the moment grouping doens't
>> >> support RankQuery.
>> >> I opened a jira item time ago
>> >> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
>> >> to receive feedback on that.  You can find the code here
>> >> https://github.com/apache/lucene-solr/pull/162.
>> >>
>> >> Cheers,
>> >> diego
>> >>
>> >> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
>> >> > Hi,
>> >> >
>> >> > I am using grouping and LTR together and the results are not getting
>> >> > re-rank as it does without grouping.
>> >> >
>> >> > I am passing &rq parameter.
>> >> >
>> >> > Does LTR work with grouping on?
>> >> > Solr version 6.5
>> >> >
>> >> > Thank you,
>> >> > Roopa
>> >>
>> >
>> >
>>
>>
>>
>
 [javac] Compiling 1089 source files to 
/Users/rrao/git/lucene-solr-all/lucene-solr/solr/build/solr-core/classes/java
[javac] 
/Users/rrao/git/lucene-solr-all/lucene-solr/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java:67:
 error: cannot find symbol
[javac] import org.apache.solr.core.SolrInfoBean;
[javac]^
[javac]   symbol:   class SolrInfoBean
[javac]   location: package org.apache.solr.core
[javac] 
/Users/rrao/git/lucene-solr-all/lucene-solr/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java:90:
 error: cannot find symbol
[javac] public class SolrIndexSearcher extends IndexSearcher implements 
Closeable, SolrInfoBean, SolrMetricProducer {
[javac]   

using rank queries(rq) with grouping in solr cloud

[tomerg]

hey,

i'm using solr 6.5.1 with solrCloud mode.
i use grouping for my results. 
i want to use rank query(rq) in order to rerank the top groups(with ltr).
it's ok for me to rerank the groups  only by reranking one of the documents
in the group. 
i saw in issue SOLR-8776 that  rank queries doesn't  support  grouping. 
(link here: https://issues.apache.org/jira/browse/SOLR-8776).

so i have a few questions:
1. there is some way to bypass this problem(or use some other existing
features of solr to achieve similar results? 
2. if there is no other way, i would like  to implement a component to
achieve this functionality(i don't want to patch the code of solr itself). 
do you have a suggestion what might be the best way to implement a rerank of
groups in cloud mode? 
can i implement something that rerank the groups for every shard before
merging  or there is a way to create component that rerank only the merged
result list from the shards? 

thanks,
tomerg



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi Diego,

Thank you,

I am interested in reranking the documents inside one of the groups.

I will try the options you mentioned here.

Thank you,
Roopa

On Mon, Dec 11, 2017 at 6:57 AM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
dceccarel...@bloomberg.net> wrote:

> Hi Roopa,
>
> If you look at the diff:
>
> https://github.com/apache/lucene-solr/pull/162/files
>
> I didn't change much in SolrIndexSearcher, you can try to skip the file
> when applying the patch and redo the changes after.
>
> Alternatively, the feature branch is available here:
>
> https://github.com/bloomberg/lucene-solr/commits/master-solr-8776
>
> you could try to merge with that or cheery-pick my changes.
>
> Are you interested in reranking the groups or also in reranking the
> documents inside each group?
>
> Cheers,
> Diego
>
>
> From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:
> solr-user@lucene.apache.org
> Subject: Re: Learning to Rank (LTR) with grouping
>
> Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
> critical.
>
> Here are the steps I took on my mac:
>
> On branch branch_6_5
>
> Your branch is up-to-date with 'origin/branch_6_5'
>
> patch -p1 -i 162.patch --dry-run
>
>
> I am getting Failures for certain Hunks
>
> Example:
>
> patching file
> solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
>
> Hunk #1 FAILED at 1471.
>
>
> Could you please give your input on how to apply this ticket as a patch for
> branch_6_5 ?
>
>
> Thank you,
>
> Roopa
>
> On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:
>
> > Hi Diego,
> >
> > Thank you, I will look into this and see how I could patch this.
> >
> > Thank you for your quick response,
> > Roopa
> >
> >
> > On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
> > diego.ceccare...@gmail.com> wrote:
> >
> >> Hi Roopa,
> >>
> >> LTR is implemented using RankQuery, and at the moment grouping doens't
> >> support RankQuery.
> >> I opened a jira item time ago
> >> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
> >> to receive feedback on that.  You can find the code here
> >> https://github.com/apache/lucene-solr/pull/162.
> >>
> >> Cheers,
> >> diego
> >>
> >> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
> >> > Hi,
> >> >
> >> > I am using grouping and LTR together and the results are not getting
> >> > re-rank as it does without grouping.
> >> >
> >> > I am passing &rq parameter.
> >> >
> >> > Does LTR work with grouping on?
> >> > Solr version 6.5
> >> >
> >> > Thank you,
> >> > Roopa
> >>
> >
> >
>
>
>

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Hi Roopa,

If you look at the diff: 

https://github.com/apache/lucene-solr/pull/162/files

I didn't change much in SolrIndexSearcher, you can try to skip the file when 
applying the patch and redo the changes after.

Alternatively, the feature branch is available here: 

https://github.com/bloomberg/lucene-solr/commits/master-solr-8776

you could try to merge with that or cheery-pick my changes. 

Are you interested in reranking the groups or also in reranking the documents 
inside each group? 

Cheers,
Diego


From: solr-user@lucene.apache.org At: 12/09/17 19:07:25To:  
solr-user@lucene.apache.org
Subject: Re: Learning to Rank (LTR) with grouping

Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
critical.

Here are the steps I took on my mac:

On branch branch_6_5

Your branch is up-to-date with 'origin/branch_6_5'

patch -p1 -i 162.patch --dry-run


I am getting Failures for certain Hunks

Example:

patching file
solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java

Hunk #1 FAILED at 1471.


Could you please give your input on how to apply this ticket as a patch for
branch_6_5 ?


Thank you,

Roopa

On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:

> Hi Diego,
>
> Thank you, I will look into this and see how I could patch this.
>
> Thank you for your quick response,
> Roopa
>
>
> On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
> diego.ceccare...@gmail.com> wrote:
>
>> Hi Roopa,
>>
>> LTR is implemented using RankQuery, and at the moment grouping doens't
>> support RankQuery.
>> I opened a jira item time ago
>> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
>> to receive feedback on that.  You can find the code here
>> https://github.com/apache/lucene-solr/pull/162.
>>
>> Cheers,
>> diego
>>
>> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
>> > Hi,
>> >
>> > I am using grouping and LTR together and the results are not getting
>> > re-rank as it does without grouping.
>> >
>> > I am passing &rq parameter.
>> >
>> > Does LTR work with grouping on?
>> > Solr version 6.5
>> >
>> > Thank you,
>> > Roopa
>>
>
>

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi I tried to apply this JIRA SOLR-8776 as a patch as this feature is
critical.

Here are the steps I took on my mac:

On branch branch_6_5

Your branch is up-to-date with 'origin/branch_6_5'

patch -p1 -i 162.patch --dry-run


I am getting Failures for certain Hunks

Example:

patching file
solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java

Hunk #1 FAILED at 1471.


Could you please give your input on how to apply this ticket as a patch for
branch_6_5 ?


Thank you,

Roopa

On Fri, Dec 8, 2017 at 6:52 PM, Roopa Rao  wrote:

> Hi Diego,
>
> Thank you, I will look into this and see how I could patch this.
>
> Thank you for your quick response,
> Roopa
>
>
> On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli <
> diego.ceccare...@gmail.com> wrote:
>
>> Hi Roopa,
>>
>> LTR is implemented using RankQuery, and at the moment grouping doens't
>> support RankQuery.
>> I opened a jira item time ago
>> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
>> to receive feedback on that.  You can find the code here
>> https://github.com/apache/lucene-solr/pull/162.
>>
>> Cheers,
>> diego
>>
>> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
>> > Hi,
>> >
>> > I am using grouping and LTR together and the results are not getting
>> > re-rank as it does without grouping.
>> >
>> > I am passing &rq parameter.
>> >
>> > Does LTR work with grouping on?
>> > Solr version 6.5
>> >
>> > Thank you,
>> > Roopa
>>
>
>

Re: Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi Diego,

Thank you, I will look into this and see how I could patch this.

Thank you for your quick response,
Roopa

On Fri, Dec 8, 2017 at 5:44 PM, Diego Ceccarelli  wrote:

> Hi Roopa,
>
> LTR is implemented using RankQuery, and at the moment grouping doens't
> support RankQuery.
> I opened a jira item time ago
> (https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
> to receive feedback on that.  You can find the code here
> https://github.com/apache/lucene-solr/pull/162.
>
> Cheers,
> diego
>
> On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
> > Hi,
> >
> > I am using grouping and LTR together and the results are not getting
> > re-rank as it does without grouping.
> >
> > I am passing &rq parameter.
> >
> > Does LTR work with grouping on?
> > Solr version 6.5
> >
> > Thank you,
> > Roopa
>

Re: Learning to Rank (LTR) with grouping

[Diego Ceccarelli]

Hi Roopa,

LTR is implemented using RankQuery, and at the moment grouping doens't
support RankQuery.
I opened a jira item time ago
(https://issues.apache.org/jira/browse/SOLR-8776) and I would be happy
to receive feedback on that.  You can find the code here
https://github.com/apache/lucene-solr/pull/162.

Cheers,
diego

On Fri, Dec 8, 2017 at 9:15 PM, Roopa Rao  wrote:
> Hi,
>
> I am using grouping and LTR together and the results are not getting
> re-rank as it does without grouping.
>
> I am passing &rq parameter.
>
> Does LTR work with grouping on?
> Solr version 6.5
>
> Thank you,
> Roopa

Learning to Rank (LTR) with grouping

[Roopa Rao]

Hi,

I am using grouping and LTR together and the results are not getting
re-rank as it does without grouping.

I am passing &rq parameter.

Does LTR work with grouping on?
Solr version 6.5

Thank you,
Roopa

RE: Result grouping performance

[Kempelen , Ákos]

Hi Mikhail,

group.facet=false, but group.truncate=true.
I found out that if the group.truncate parameter is true, the query will be 
much slower (x5 times), regardless of faceting.
I thought, that the group.truncate param is meaningless without activating the 
facet system, but this is not the case.

Testing environment: Solr 7.0.1, 1 core with 70GB data (~1.000.000 documents), 
no shards

Akos



-Original Message-
From: Mikhail Khludnev [mailto:m...@apache.org] 
Sent: Thursday, November 23, 2017 8:38 AM
To: solr-user 
Subject: Re: Result grouping performance

Akos,
Can you provide your request params? Do you just group and/or count grouped 
facets?
Can you clarify how field collapsing is different from grouping, just make it 
unambiguous?


On Wed, Nov 22, 2017 at 4:13 PM, Kempelen, Ákos < 
akos.kempe...@wolterskluwer.com> wrote:

> Hello,
>
> I am migrating our codebase from Solr 4.7 to 7.0.1 but the performance 
> of result grouping seems very poor using the newer Solr.
> For example a simple MatchAllDocsQuery takes 5 sec on Solr4.7, and 21 
> sec on Solr7.
> I wonder what causes the x4 difference in time? We hoped that newer 
> Solr versions will provide better performances...
> Using Field collapsing could would be a solution, but it produces 
> different facet counts.
> Thanks,
> Akos
>
>
>


--
Sincerely yours
Mikhail Khludnev

Re: Result grouping performance

[Mikhail Khludnev]

Akos,
Can you provide your request params? Do you just group and/or count grouped
facets?
Can you clarify how field collapsing is different from grouping, just make
it unambiguous?


On Wed, Nov 22, 2017 at 4:13 PM, Kempelen, Ákos <
akos.kempe...@wolterskluwer.com> wrote:

> Hello,
>
> I am migrating our codebase from Solr 4.7 to 7.0.1 but the performance of
> result grouping seems very poor using the newer Solr.
> For example a simple MatchAllDocsQuery takes 5 sec on Solr4.7, and 21 sec
> on Solr7.
> I wonder what causes the x4 difference in time? We hoped that newer Solr
> versions will provide better performances...
> Using Field collapsing could would be a solution, but it produces
> different facet counts.
> Thanks,
> Akos
>
>
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Result grouping performance

[Erick Erickson]

Have you enabled docValues (and reindexed from scratch) on the field
you're grouping on?

Best,
Erick

On Wed, Nov 22, 2017 at 5:13 AM, Kempelen, Ákos
 wrote:
> Hello,
>
> I am migrating our codebase from Solr 4.7 to 7.0.1 but the performance of 
> result grouping seems very poor using the newer Solr.
> For example a simple MatchAllDocsQuery takes 5 sec on Solr4.7, and 21 sec on 
> Solr7.
> I wonder what causes the x4 difference in time? We hoped that newer Solr 
> versions will provide better performances...
> Using Field collapsing could would be a solution, but it produces different 
> facet counts.
> Thanks,
> Akos
>
>

Result grouping performance

[Kempelen , Ákos]

Hello,

I am migrating our codebase from Solr 4.7 to 7.0.1 but the performance of 
result grouping seems very poor using the newer Solr.
For example a simple MatchAllDocsQuery takes 5 sec on Solr4.7, and 21 sec on 
Solr7.
I wonder what causes the x4 difference in time? We hoped that newer Solr 
versions will provide better performances...
Using Field collapsing could would be a solution, but it produces different 
facet counts.
Thanks,
Akos 

Re: Grouping and group.facet performance disaster

[Ere Maijala]

While I can't say whether it affects you in this case, Solr 6.4.1 has 
serious performance issues. I'd suggest upgrading to at least 6.4.2.


--Ere

31.5.2017, 14.16, Marek Tichy kirjoitti:

Hi,

I'm getting a very slow response times on grouping, especially on facet
grouping.

Without grouping, the query takes 14ms, faceting 57ms.

With grouping, the query time goes up to 1131ms, with facet grouping,
the faceting goes up to the unbearable 12103 ms.

Single solr instance, 927086docs, 518.23 MB size, solr 6.4.1.

Is this really the price of grouping ? Are there any magic
tricks/tips/techniques to improve the speed ?
The query params below.

Many thanks for any help, much appreciated.

Best
 Marek Tichy








fq=((type:knihy) OR (type:defekty))
fl=*
start=0
f.ebook_formats.facet.mincount=1
f.authorid.facet.mincount=1
f.thematicgroupid.facet.mincount=1
f.articleparts.facet.mincount=1
f.type.facet.mincount=1
f.languageid.facet.mincount=1
f.showwindow.facet.mincount=1
f.articletypeid_grouped.facet.mincount=1
f.languageid.facet.limit=10
f.ebook_formats.facet.limit=10
f.authorid.facet.limit=10
f.type.facet.limit=10
f.articleparts.facet.limit=10
f.thematicgroupid.facet.limit=10
f.articletypeid_grouped.facet.limit=10
f.showwindow.facet.limit=100
version=2.2
group.limit=30
rows=30
echoParams=all
sort=date desc,planneddate asc
group.field=edition
facet.method=enum
group.truncate=false
group.format=grouped
group=true
group.ngroups=true
stats=true
facet=true
group.facet=true
stats.field={!distinctValues=true}categoryid
facet.field={!ex=at}articletypeid_grouped
facet.field={!ex=at}type
facet.field={!ex=author}authorid
facet.field={!ex=format}articleparts
facet.field={!ex=format}ebook_formats
facet.field={!ex=lang}languageid
facet.field={!ex=sw}showwindow
facet.field={!ex=tema}thematicgroupid
stats.field={!min=true max=true}price
stats.field={!min=true max=true}yearout



--
Ere Maijala
Kansalliskirjasto / The National Library of Finland

Re: [EXTERNAL] Grouping and group.facet performance disaster

[Sunil . Srinivasan]

Use group.cache.percent – for your index size, it might work well.

Thanks,

On 5/31/17, 4:16 AM, "Marek Tichy"  wrote:

Hi,

I'm getting a very slow response times on grouping, especially on facet
grouping.
    
Without grouping, the query takes 14ms, faceting 57ms.

    With grouping, the query time goes up to 1131ms, with facet grouping,
the faceting goes up to the unbearable 12103 ms.

Single solr instance, 927086docs, 518.23 MB size, solr 6.4.1.

Is this really the price of grouping ? Are there any magic
tricks/tips/techniques to improve the speed ?
The query params below.

Many thanks for any help, much appreciated.

Best
 Marek Tichy








fq=((type:knihy) OR (type:defekty))
fl=*
start=0
f.ebook_formats.facet.mincount=1
f.authorid.facet.mincount=1
f.thematicgroupid.facet.mincount=1
f.articleparts.facet.mincount=1
f.type.facet.mincount=1
f.languageid.facet.mincount=1
f.showwindow.facet.mincount=1
f.articletypeid_grouped.facet.mincount=1
f.languageid.facet.limit=10
f.ebook_formats.facet.limit=10
f.authorid.facet.limit=10
f.type.facet.limit=10
f.articleparts.facet.limit=10
f.thematicgroupid.facet.limit=10
f.articletypeid_grouped.facet.limit=10
f.showwindow.facet.limit=100
version=2.2
group.limit=30
rows=30
echoParams=all
sort=date desc,planneddate asc
group.field=edition
facet.method=enum
group.truncate=false
group.format=grouped
group=true
group.ngroups=true
stats=true
facet=true
group.facet=true
stats.field={!distinctValues=true}categoryid
facet.field={!ex=at}articletypeid_grouped
facet.field={!ex=at}type
facet.field={!ex=author}authorid
facet.field={!ex=format}articleparts
facet.field={!ex=format}ebook_formats
facet.field={!ex=lang}languageid
facet.field={!ex=sw}showwindow
facet.field={!ex=tema}thematicgroupid
stats.field={!min=true max=true}price
stats.field={!min=true max=true}yearout

Re: Grouping and group.facet performance disaster

[Susheel Kumar]

Did you try sub-facets ( http://yonik.com/json-facet-api/ )  if that meets
your facet grouping requirements or try Collapse/Expand Results.
https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results


Thnx

On Wed, May 31, 2017 at 7:16 AM, Marek Tichy  wrote:

> Hi,
>
> I'm getting a very slow response times on grouping, especially on facet
> grouping.
>
> Without grouping, the query takes 14ms, faceting 57ms.
>
> With grouping, the query time goes up to 1131ms, with facet grouping,
> the faceting goes up to the unbearable 12103 ms.
>
> Single solr instance, 927086docs, 518.23 MB size, solr 6.4.1.
>
> Is this really the price of grouping ? Are there any magic
> tricks/tips/techniques to improve the speed ?
> The query params below.
>
> Many thanks for any help, much appreciated.
>
> Best
>  Marek Tichy
>
>
>
>
>
>
>
>
> fq=((type:knihy) OR (type:defekty))
> fl=*
> start=0
> f.ebook_formats.facet.mincount=1
> f.authorid.facet.mincount=1
> f.thematicgroupid.facet.mincount=1
> f.articleparts.facet.mincount=1
> f.type.facet.mincount=1
> f.languageid.facet.mincount=1
> f.showwindow.facet.mincount=1
> f.articletypeid_grouped.facet.mincount=1
> f.languageid.facet.limit=10
> f.ebook_formats.facet.limit=10
> f.authorid.facet.limit=10
> f.type.facet.limit=10
> f.articleparts.facet.limit=10
> f.thematicgroupid.facet.limit=10
> f.articletypeid_grouped.facet.limit=10
> f.showwindow.facet.limit=100
> version=2.2
> group.limit=30
> rows=30
> echoParams=all
> sort=date desc,planneddate asc
> group.field=edition
> facet.method=enum
> group.truncate=false
> group.format=grouped
> group=true
> group.ngroups=true
> stats=true
> facet=true
> group.facet=true
> stats.field={!distinctValues=true}categoryid
> facet.field={!ex=at}articletypeid_grouped
> facet.field={!ex=at}type
> facet.field={!ex=author}authorid
> facet.field={!ex=format}articleparts
> facet.field={!ex=format}ebook_formats
> facet.field={!ex=lang}languageid
> facet.field={!ex=sw}showwindow
> facet.field={!ex=tema}thematicgroupid
> stats.field={!min=true max=true}price
> stats.field={!min=true max=true}yearout
>

Grouping and group.facet performance disaster

[Marek Tichy]

Hi,

I'm getting a very slow response times on grouping, especially on facet
grouping.

Without grouping, the query takes 14ms, faceting 57ms.

With grouping, the query time goes up to 1131ms, with facet grouping,
the faceting goes up to the unbearable 12103 ms.

Single solr instance, 927086docs, 518.23 MB size, solr 6.4.1.

Is this really the price of grouping ? Are there any magic
tricks/tips/techniques to improve the speed ?
The query params below.

Many thanks for any help, much appreciated.

Best
 Marek Tichy








fq=((type:knihy) OR (type:defekty))
fl=*
start=0
f.ebook_formats.facet.mincount=1
f.authorid.facet.mincount=1
f.thematicgroupid.facet.mincount=1
f.articleparts.facet.mincount=1
f.type.facet.mincount=1
f.languageid.facet.mincount=1
f.showwindow.facet.mincount=1
f.articletypeid_grouped.facet.mincount=1
f.languageid.facet.limit=10
f.ebook_formats.facet.limit=10
f.authorid.facet.limit=10
f.type.facet.limit=10
f.articleparts.facet.limit=10
f.thematicgroupid.facet.limit=10
f.articletypeid_grouped.facet.limit=10
f.showwindow.facet.limit=100
version=2.2
group.limit=30
rows=30
echoParams=all
sort=date desc,planneddate asc
group.field=edition
facet.method=enum
group.truncate=false
group.format=grouped
group=true
group.ngroups=true
stats=true
facet=true
group.facet=true
stats.field={!distinctValues=true}categoryid
facet.field={!ex=at}articletypeid_grouped
facet.field={!ex=at}type
facet.field={!ex=author}authorid
facet.field={!ex=format}articleparts
facet.field={!ex=format}ebook_formats
facet.field={!ex=lang}languageid
facet.field={!ex=sw}showwindow
facet.field={!ex=tema}thematicgroupid
stats.field={!min=true max=true}price
stats.field={!min=true max=true}yearout

Re: Pagination issue when grouping

[Rick Leir]

Hi Tien
Consider using the export handler if you can. Then you have no paging.

When you are having a paging problem you might want to think of the use case -- 
how many of your users will be willing​ to page deeply? If they give up, then 
you have lost already.
Cheers -- Rick

On May 29, 2017 7:58:38 PM EDT, Nguyen Manh Tien  
wrote:
>Hello,
>
>I group search result by a field (with high cardinality)
>I paginate search page using num of groups using param
>group.ngroups=true.
>But that cause high CPU issue. So i turn off it.
>
>Without ngroups=true, i can't get the num of groups so pagination is
>not
>correct because i must use numFound,
>
>it alway miss some last pages, the reason is some results was already
>collapsed into groups in previous pages.
>
>For example, a search return 11 results, but there are 2 results belong
>to
>1 groups, so it has 10 groups (but i don't know it in advance because i
>set
>ngroups=false), with 11 results, pagination display 2 pages, but page 2
>have 0 results.
>
>Anyone faced similar issue and had a work around?
>
>Thanks,
>Tien

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Pagination issue when grouping

[Nguyen Manh Tien]

Hello,

I group search result by a field (with high cardinality)
I paginate search page using num of groups using param group.ngroups=true.
But that cause high CPU issue. So i turn off it.

Without ngroups=true, i can't get the num of groups so pagination is not
correct because i must use numFound,

it alway miss some last pages, the reason is some results was already
collapsed into groups in previous pages.

For example, a search return 11 results, but there are 2 results belong to
1 groups, so it has 10 groups (but i don't know it in advance because i set
ngroups=false), with 11 results, pagination display 2 pages, but page 2
have 0 results.

Anyone faced similar issue and had a work around?

Thanks,
Tien

Re: Grouping by a multivalued field

[Rick Leir]

Shacky
Quote "A multivalued field is useful when there are more than one value present 
for the field. An easy example would be tags, there can be multiple tags that 
need to be indexed...". So yes, you are on the right track. Cheers -- Rick
https://stackoverflow.com/questions/5800762/what-is-the-use-of-multivalued-field-type-in-solr


On May 26, 2017 9:45:48 AM EDT, shacky  wrote:
>Hi,
>I need to create a new collection on my Solr 6.1.0 cluster where every
>row
>is a "content" and every content can belong to one or many categories,
>which are specified in a multivalued field "categories".
>
>In my web app the user can search by categories, and if wanted it can
>even
>group results by category. If it wants to group by category, what about
>the
>contents which belongs to more than one category?
>
>In this case the search results page should show the same content more
>times in different categories. I don't want the web application to
>filter
>and order results because in this case it should ask Solr for every
>rows (I
>know this is not advised for bad performance), so is there a way to let
>Solr make this? For example, repeating the same content in two
>categories
>if a flag is enabled or if I am asking Solr to sort by category?
>
>Thank you very much!
>Bye

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Grouping by a multivalued field

[shacky]

Hi,
I need to create a new collection on my Solr 6.1.0 cluster where every row
is a "content" and every content can belong to one or many categories,
which are specified in a multivalued field "categories".

In my web app the user can search by categories, and if wanted it can even
group results by category. If it wants to group by category, what about the
contents which belongs to more than one category?

In this case the search results page should show the same content more
times in different categories. I don't want the web application to filter
and order results because in this case it should ask Solr for every rows (I
know this is not advised for bad performance), so is there a way to let
Solr make this? For example, repeating the same content in two categories
if a flag is enabled or if I am asking Solr to sort by category?

Thank you very much!
Bye

Re: High CPU when use grouping group.ngroups=true

[Nguyen Manh Tien]

Without using ngroups=true, is there any way to handle pagination correctly
when we collapse result using grouping?

Regards,
Tien

On Tue, May 23, 2017 at 9:55 PM, Nguyen Manh Tien  wrote:

> The collapse field is high-cardinality field. I haven't profiling yet but
> will do it.
>
> Thanks,
> Tien
>
> On Tue, May 23, 2017 at 9:48 PM, Erick Erickson 
> wrote:
>
>> How many unique values in your group field? For high-cardinality
>> fields there's quite a bit of bookkeeping that needs to be done.
>>
>> Have you tried profiling to see where the CPU time is being spent?
>>
>> Best,
>> Erick
>>
>> On Tue, May 23, 2017 at 7:46 AM, Nguyen Manh Tien
>>  wrote:
>> > Hi All,
>> >
>> > I recently switch from solr field collapse/expand to grouping for
>> collapse
>> > search result
>> > All seem good but CPU is always high (80-100%) when i set param
>> > group.ngroups=true.
>> >
>> > We set ngroups=true to get number of groups so that we can paginate
>> search
>> > result correctly.
>> > Due to CPU issue we need to turn it off.
>> >
>> > Is ngroups=true is expensive feature? Is there any way to prevent CPU
>> issue
>> > and still have correct pagination.
>> >
>> > Thanks,
>> > Tien
>>
>
>

Re: High CPU when use grouping group.ngroups=true

[Nguyen Manh Tien]

The collapse field is high-cardinality field. I haven't profiling yet but
will do it.

Thanks,
Tien

On Tue, May 23, 2017 at 9:48 PM, Erick Erickson 
wrote:

> How many unique values in your group field? For high-cardinality
> fields there's quite a bit of bookkeeping that needs to be done.
>
> Have you tried profiling to see where the CPU time is being spent?
>
> Best,
> Erick
>
> On Tue, May 23, 2017 at 7:46 AM, Nguyen Manh Tien
>  wrote:
> > Hi All,
> >
> > I recently switch from solr field collapse/expand to grouping for
> collapse
> > search result
> > All seem good but CPU is always high (80-100%) when i set param
> > group.ngroups=true.
> >
> > We set ngroups=true to get number of groups so that we can paginate
> search
> > result correctly.
> > Due to CPU issue we need to turn it off.
> >
> > Is ngroups=true is expensive feature? Is there any way to prevent CPU
> issue
> > and still have correct pagination.
> >
> > Thanks,
> > Tien
>

Re: High CPU when use grouping group.ngroups=true

[Erick Erickson]

How many unique values in your group field? For high-cardinality
fields there's quite a bit of bookkeeping that needs to be done.

Have you tried profiling to see where the CPU time is being spent?

Best,
Erick

On Tue, May 23, 2017 at 7:46 AM, Nguyen Manh Tien
 wrote:
> Hi All,
>
> I recently switch from solr field collapse/expand to grouping for collapse
> search result
> All seem good but CPU is always high (80-100%) when i set param
> group.ngroups=true.
>
> We set ngroups=true to get number of groups so that we can paginate search
> result correctly.
> Due to CPU issue we need to turn it off.
>
> Is ngroups=true is expensive feature? Is there any way to prevent CPU issue
> and still have correct pagination.
>
> Thanks,
> Tien

High CPU when use grouping group.ngroups=true

[Nguyen Manh Tien]

Hi All,

I recently switch from solr field collapse/expand to grouping for collapse
search result
All seem good but CPU is always high (80-100%) when i set param
group.ngroups=true.

We set ngroups=true to get number of groups so that we can paginate search
result correctly.
Due to CPU issue we need to turn it off.

Is ngroups=true is expensive feature? Is there any way to prevent CPU issue
and still have correct pagination.

Thanks,
Tien

Support RankQuery in grouping

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Hi All, 
At the moment RankQueries [1]  are not supported when you perform grouping: 
if you perform a ReRankQuery and ask for the groups, reranking will be ignored 
in the scoring. 
In SOLR-8776, I added support for ReRankQueries in grouping and I opened a PR 
on github [2]. 
ReRankQueries are now supported in standalone and distribute mode, and I 
changed 
the logic to rerank groups and not only documents (see the Jira item [3] for 
more details). 

IMO the patch is now quite complete and I've working unit tests, so I was 
wondering if someone could review it :) 
Thanks,
Diego

[1] https://cwiki.apache.org/confluence/display/solr/Query+Re-Ranking   
[2] https://github.com/apache/lucene-solr/pull/162
[3] https://issues.apache.org/jira/browse/SOLR-8776

Re: Search inside grouping list

[Emir Arnautovic]

Can you try reproducing this issue on fresh Solr, and if you manage to, 
can you please share documents and steps to reproduce it.


Which version of Solr do you run and do you have any custom plugins 
running on it?


Emir


On 09.05.2017 13:01, donjose wrote:

Yes. I am getting the same result for both q and  fq



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4334206.html
Sent from the Solr - User mailing list archive at Nabble.com.


--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Re: Search inside grouping list

[donjose]

Yes. I am getting the same result for both q and  fq



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4334206.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Search inside grouping list

[Emir Arnautovic]

Do you get the same result if you use q instead of fq?


On 09.05.2017 07:38, donjose wrote:

Hi Emir,

Grouping by default is part of the configuration


  true
  assetid
  true


Don.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4334136.html
Sent from the Solr - User mailing list archive at Nabble.com.


--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Re: Search inside grouping list

[donjose]

Hi Emir,

Grouping by default is part of the configuration

   
 true
 assetid
 true 
   

Don.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4334136.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Search inside grouping list

[Emir Arnautovic]

Hi Don,

This is query without grouping and returns expected results. But when 
you apply grouping by some field, you get wrong results? Can you share 
query results and query with grouping.


Emir


On 08.05.2017 14:28, donjose wrote:

Hi Emir,
Thank you for the response.

Please find the query which i am sending to SOLR
http://localhost:8983/solr/pema/select?fq=color:red&indent=on&q=*:*&wt=json

Regards,
Don.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4333936.html
Sent from the Solr - User mailing list archive at Nabble.com.


--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Re: Search inside grouping list

[donjose]

Hi Emir,
Thank you for the response.

Please find the query which i am sending to SOLR
http://localhost:8983/solr/pema/select?fq=color:red&indent=on&q=*:*&wt=json

Regards,
Don.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4333936.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Search inside grouping list

[Emir Arnautovic]

Hi,

Can you please provide full query that you are sending to Solr.

Thanks,
Emir


On 08.05.2017 07:18, donjose wrote:

Could anyone can please reply for this query



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-inside-grouping-list-tp4333488p4333870.html
Sent from the Solr - User mailing list archive at Nabble.com.


--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/