@florin Great advice. Null key for unique documents is really helpful. Any other such tricks that you are using to improve collapse performance ?
On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar <parshant.ku...@indiamart.com.invalid> wrote: > Hi Joel, > > 1) What are the response times for both methods. Saying one is faster is > not specific enough. > > Response time for the grouped method is 167 ms for 0.65 million requests. > Response time for the collapsed method is 177 ms for 0.65 million requests. > > 2) What is the cardinality of the collapse field, saying it's high is not > specific enough. What is the actual cardinality? > > Cardinality of the collapse field is around 6.2 Million > > [image: image.png] > 3) Is ngroups used in the grouping query > > Yes, ngroups is used in grouping query. > > Thanks > Parshant Kumar > > > > > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <joels...@gmail.com> wrote: > >> Collapse is designed to outperform grouping in the following scenario: >> >> There is high cardinality on the group field and group.ngroups is needed. >> If either of these conditions is not satisfied grouping will typically be >> faster. >> >> You will need to provide some more information about your setup to get an >> answer to the collapse performance question. >> >> 1) What are the response times for both methods. Saying one is faster is >> not specific enough. >> 2) What is the cardinality of the collapse field, saying it's high is not >> specific enough. What is the actual cardinality? >> 3) Is ngroups used in the grouping query. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Joel Bernstein >> http://joelsolr.blogspot.com/ >> >> >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <gajju3...@gmail.com> >> wrote: >> >> > @prashant Florin means to put null for parentglusrid in documents where >> > this field-value is only present in one document [Group has only one >> > document]. and then use nullPolicy to include/expand. >> > >> > >> > >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar >> > <parshant.ku...@indiamart.com.invalid> wrote: >> > >> > > client should set to null the field if it's unique. >> > > >> > > @florin @Gajendra can you please explain more .I am not clear how to >> > > perform this. >> > > >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <babesflo...@gmail.com> >> > wrote: >> > > >> > > > @Gajendra Our response time dropped by 36% and our rps increased >> with >> > > 27%. >> > > > >> > > > You have to reindex the core and the client should set to null the >> > field >> > > if >> > > > it's unique. >> > > > >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar >> > > > <parshant.ku...@indiamart.com.invalid> a scris: >> > > > >> > > > > How can we make group_field null? Using nullPolicy=expand ? >> > > > > >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes < >> babesflo...@gmail.com> >> > > > wrote: >> > > > > >> > > > > > We improved the performance of collapse by making the >> group_field >> > > null >> > > > > for >> > > > > > the documents that have an unique value for group_field. This >> might >> > > > help/ >> > > > > > >> > > > > > >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar >> > > > > > <parshant.ku...@indiamart.com.invalid> a scris: >> > > > > > >> > > > > > > yes,group_field is having high cardinality. >> > > > > > > >> > > > > > > >> > > > > > > Thanks >> > > > > > > Parshant Kumar >> > > > > > > >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes < >> > babesflo...@gmail.com >> > > > >> > > > > > wrote: >> > > > > > > >> > > > > > > > Your group_field has a high cardinality? >> > > > > > > > Thanks, >> > > > > > > > Florin Babes >> > > > > > > > >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar >> > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris: >> > > > > > > > >> > > > > > > > > Hi florin, >> > > > > > > > > >> > > > > > > > > I am using below. >> > > > > > > > > >> > > > > > > > > 1) fq={!collapse field=parentglusrid} >> > > > > > > > > 2) expand.rows=4 >> > > > > > > > > 3) expand=true >> > > > > > > > > >> > > > > > > > > Size of index is around 100GB. >> > > > > > > > > Solr version is 6.5 >> > > > > > > > > >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes < >> > > > babesflo...@gmail.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hello, >> > > > > > > > > > First let's call the field you collapse on group_field >> > > > > > > > > > If group_field has a high cardinality you should make >> > > > group_field >> > > > > > > null >> > > > > > > > > for >> > > > > > > > > > those documents that have a unique group_field and set >> > > > > > > > nullPolicy=expand. >> > > > > > > > > > By doing that solr will use less memory for it's >> internal >> > > maps >> > > > > (so >> > > > > > > > faster >> > > > > > > > > > gc) and the head selecting will be faster. >> > > > > > > > > > What is your head selecting strategy? Can you share >> your fq >> > > > which >> > > > > > you >> > > > > > > > use >> > > > > > > > > > for collapsing? >> > > > > > > > > > >> > > > > > > > > > Thanks, >> > > > > > > > > > Florin Babes >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar >> > > > > > > > > > <parshant.ku...@indiamart.com.invalid> a scris: >> > > > > > > > > > >> > > > > > > > > > > anyone please help >> > > > > > > > > > > >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar < >> > > > > > > > > > > parshant.ku...@indiamart.com> >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi all, >> > > > > > > > > > > > >> > > > > > > > > > > > We have implemented collapse queries in place of >> > grouped >> > > > > > queries >> > > > > > > on >> > > > > > > > > our >> > > > > > > > > > > > production solr. As mentioned in solr documentation >> > > > collapse >> > > > > > > > queries >> > > > > > > > > > are >> > > > > > > > > > > > recommended in place of grouped queries in terms of >> > > > > > performance . >> > > > > > > > But >> > > > > > > > > > > after >> > > > > > > > > > > > switching to collapsed queries from grouped queries >> > > > response >> > > > > > time >> > > > > > > > of >> > > > > > > > > > > > queries have increased. This is unexpected >> behaviour, >> > the >> > > > > > > response >> > > > > > > > > time >> > > > > > > > > > > > should have been improved but results are opposites. >> > > > > > > > > > > > Please someone help why response time is increased >> for >> > > > > > collapsed >> > > > > > > > > > queries. >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks >> > > > > > > > > > > > Parshant Kumar >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > -- >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > -- >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > >> > > >> > > -- >> > > >> > > >> > >> > >
