Re: Sorting on pseudo field(The one which is added during doctransformer)

[prateek . agarwal]

Thanks, Mikhail

On 2018/05/18 11:33:09, Mikhail Khludnev  wrote: 
> Right
> https://wiki.apache.org/solr/SolrPlugins#ValueSourceParser
> 
> On Fri, May 18, 2018 at 8:04 AM, prateek.agar...@bigbasket.com <
> prateek.agar...@bigbasket.com> wrote:
> 
> > Hi Mikhail,
> >
> > I think you forgot to link the reference.
> >
> > Thanks
> >
> >
> > Regards,
> > Prateek
> >
> > On 2018/05/17 13:18:22, Mikhail Khludnev  wrote:
> > > Here is the reference I've found so far.
> > >
> > > On Thu, May 17, 2018 at 12:26 PM, prateek.agar...@bigbasket.com <
> > > prateek.agar...@bigbasket.com> wrote:
> > >
> > > >
> > > > Hi Mikhail,
> > > >
> > > > > You can either sort by function that needs to turn the logic into
> > value
> > > > > source parser.
> > > >
> > > > But like my requirement for this was to add a field dynamically from
> > cache
> > > > or external source to the returned documents from the solr and perform
> > > > sorting in the solr itself if required otherwise use the score to sort.
> > > > So how would you advise to go about this??
> > > >
> > > > And how to go about your way "to turn the logic into value source
> > parser"
> > > > like how to do this for this case??
> > > >
> > > >
> > > > > If you need to toss just result page, check rerank.
> > > >
> > > > I don't want to use it to rank the relevancy of results.
> > > >
> > > > Thanks for the response.
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Prateek
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
> 
> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev
> 

Re: Sorting on pseudo field(The one which is added during doctransformer)

[prateek . agarwal]

Hi Mikhail,

I think you forgot to link the reference.

Thanks


Regards,
Prateek

On 2018/05/17 13:18:22, Mikhail Khludnev  wrote: 
> Here is the reference I've found so far.
> 
> On Thu, May 17, 2018 at 12:26 PM, prateek.agar...@bigbasket.com <
> prateek.agar...@bigbasket.com> wrote:
> 
> >
> > Hi Mikhail,
> >
> > > You can either sort by function that needs to turn the logic into value
> > > source parser.
> >
> > But like my requirement for this was to add a field dynamically from cache
> > or external source to the returned documents from the solr and perform
> > sorting in the solr itself if required otherwise use the score to sort.
> > So how would you advise to go about this??
> >
> > And how to go about your way "to turn the logic into value source parser"
> > like how to do this for this case??
> >
> >
> > > If you need to toss just result page, check rerank.
> >
> > I don't want to use it to rank the relevancy of results.
> >
> > Thanks for the response.
> >
> >
> >
> > Regards,
> > Prateek
> >
> 
> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev
> 

Re: Sorting on pseudo field(The one which is added during doctransformer)

[prateek . agarwal]

Hi Mikhail,

> You can either sort by function that needs to turn the logic into value
> source parser.

But like my requirement for this was to add a field dynamically from cache or 
external source to the returned documents from the solr and perform sorting in 
the solr itself if required otherwise use the score to sort.
So how would you advise to go about this??

And how to go about your way "to turn the logic into value source parser" like 
how to do this for this case??


> If you need to toss just result page, check rerank.

I don't want to use it to rank the relevancy of results.

Thanks for the response.



Regards,
Prateek

Sorting on pseudo field(The one which is added during doctransformer)

[prateek . agarwal]

Hi all,

I wanted to ask is it possible to sort on the field which is added during 
DocTransformer. 
I'm referring to something like this: 
(https://mariofebbraio.files.wordpress.com/2014/10/doctransformer.png) here the 
price is added using doctransformer so is it possible to sort on this field if 
I add that field dynamically to the every returned document. If Yes, Can you 
please guide me like how to go about it.

Can anyone also tell me like at what stage sorting is performed after 
doctransformer or after the requestHandler??

Sorry if you find these queries silly but I'm really confused where all this 
happens.

Thanks in advance.


Regards,
Prateek

Re: Regarding LTR feature

[prateek . agarwal]

Hi Alessandro,

You're right it doesn't have to be that accurate to the query time but our 
requirement is having a more solid control over our outputs from Solr like if 
we have 4 features then we can adjust the weights giving something like 
(40,20,20,20) to each feature such that the sum total of features for a 
document is 100 this is only possible if we could scale the feature outputs 
between 0-1.

Secondly, I also have a doubt regarding the scaling function like why it is not 
considering only the documents filtered out by the FQ filter and considering 
all the documents which match the query.

Thaks a lot in advance.
Looking forward to hearing back from you soon.


Regards,
Prateek

On 2018/05/04 10:26:55, Alessandro Benedetti  wrote: 
> Hi Preteek,
> I would assume you have that feature at training time as well, can't you use
> the training set to estabilish the parameters for the normalizer at query
> time ?
> 
> In the end being a normalization, doesn't have to be that accurate to the
> query time state, but it must reflect the relations the model learnt from
> the training set.
> Let me know !
> 
> 
> 
> -
> ---
> Alessandro Benedetti
> Search Consultant, R Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> 

Re: Regarding LTR feature

[prateek . agarwal]

Thanks again Alessandro

I tried with the feature and the Minmax normalizer you told.But then there is a 
slight problem with the params in normalization. I don't really know the 
range(Min, Max) of values the payload_score outputs and they are different for 
different queries.

I even tried looking at the source code to see if there is a way I can override 
a class so that it iterates over all the re-ranked documents and calculate Max 
and min there itself and pass it to MinMax normalizer class but it seems it's 
not possible.

Your help will really appreciated.

Thanks



Regards,
Prateek



On 2018/05/03 14:00:00, Alessandro Benedetti  wrote: 
> Mmmm, first of all, you know that each Solr feature is calculated per
> document right ?
> So you want to calculate the payload score for the document you are
> re-ranking, based on the query ( your External Feature Information) and
> normalize across the different documents?
> 
> I would go with this feature and use the normalization LTR functionality :
> 
> { 
>   "store" : "my_feature_store", 
>   "name" : "in_aggregated_terms", 
>   "class" : "org.apache.solr.ltr.feature.SolrFeature", 
>   "params" : { "q" : "{!payload_score 
> f=aggregated_terms func=max v=${query}}" } 
> } 
> 
> Then in the model you specify something like :
> 
> "name" : "myModelName",
>"features" : [
>{
>  "name" : "isBook"
>},
> ...
>{
>  "name" : "in_aggregated_terms",
>  "norm": {
>  "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
>  "params" : { "min":"x", "max":"y" }
>  }
>},
>}
> 
> Give it a try, let me know
> 
> 
> 
> 
> -
> ---
> Alessandro Benedetti
> Search Consultant, R Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> 

Re: Regarding LTR feature

[prateek . agarwal]

Thanks again Alessandro

I tried with the feature and the Minmax normalizer you told.But then there is a 
slight problem with the params in normalization. I don't really know the 
range(Min, Max) of values the payload_score outputs and they are different for 
different queries.

I even tried looking at the source code to see if there is a way I can override 
a class so that it iterates over all the re-ranked documents and calculate Max 
and min there itself and pass it to MinMax normalizer class but it seems it's 
not possible.

Your help will be really appreciated.

Thanks



Regards,
Prateek

Re: Regarding LTR feature

[prateek . agarwal]

Hi Alessandro,

Thanks for responding.

Let me take a step back and tell you the problem I have been facing with
this.So one of the features in my LTR model is:

{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
}

so now with this feature if i apply FQ in solr it will scale the
values for all the documents irrespective of the FQ filter.

But if I change the feature to something like this:

{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!field f=aggregated_terms
v=${query}}),0,100)" }
}

Then the it scales properly with FQ aswell.

And about that verification I simply check the results returned like
in Case 1 after applying the FQ filter that feature score doesn't
scale to its maximum value of 100 which i think is because of the fact
that it scales over all the documents and returns only the subset with
the FQ filter applied.

Alternatively is their any way I can scale these value during
normalization time with a customized class which iterates over all the
re-ranked documents only.

Thanks a lot in advance.

Looking forward to hearing back from you soon.


Regards,

Prateek

Re: Regarding LTR feature

[Prateek Agarwal]

Hi Alessandro,

Thanks for responding.

Let me take a step back and tell you the problem I have been facing with
this.So one of the features in my LTR model is:

{
  "store" : "my_feature_store",
  "name" : "in_aggregated_terms",
  "class" : "org.apache.solr.ltr.feature.SolrFeature",
  "params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
}

so now with this feature if i apply FQ in solr it will scale the
values for all the documents irrespective of the FQ filter.

But if I change the feature to something like this:

{
  "store" : "my_feature_store",
  "name" : "in_aggregated_terms",
  "class" : "org.apache.solr.ltr.feature.SolrFeature",
  "params" : { "q" : "{!func}scale(query({!field f=aggregated_terms
v=${query}}),0,100)" }
}

Then the it scales properly with FQ aswell.

And about that verification I simply check the results returned like
in Case 1 after applying the FQ filter that feature score doesn't
scale to its maximum value of 100 which i think is because of the fact
that it scales over all the documents and returns only the subset with
the FQ filter applied.

Alternatively is their any way I can scale these value during
normalization time with a customized class which iterates over all the
re-ranked documents only.

Thanks a lot in advance.

Looking forward to hearing back from you soon.


Regards,

Prateek

Regarding LTR feature

[Prateek Agarwal]

Hi all,

I'm new to solr ltr and stuck on this problem for a while.

I wanted to ask why the documents on which the ltr feature score is
calculated doesn't filter out the documents even if we provide the fq
filter in the url like:

=juice={!ltr%20model=my_feature_model%20efi.query=$q%
20reRankDocs=300%20efi.store=1}=parent_store_3630_i:%201

Here the score calculation for features should only use the documents
returned from these fq parameter but it's not really the case. Is it a bug
or something.

Thanks in advance.


Regards,
Prateek

Query regarding LTR plugin in solr

[prateek . agarwal]

Hi,

I'm working on ltr feature in solr. I have a feature like :
''' {
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score f=aggregated_terms 
func=max v=${query}}),0,100)" }
  } '''

Here the scaling function is taking a lot more time than expected. 
Is there a way I could implement a customized class or any other way by which I 
can reduce this time.

So basically I just want to scale the value which looks at the whole result set 
instead of just the current document. Can I have/implement something during 
normalization??


Thanks in advance


Regards,
Prateek

Query regarding LTR plugin in solr

[Prateek Agarwal]

Hi,

I'm working on ltr feature in solr. I have a feature like :
''' {
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
  } '''

Here the scaling function is taking a lot more time than expected.
Is there a way I could implement a customized class or any other way by
which I can reduce this time.

So basically I just want to scale the value which looks at the whole result
set instead of just the current document. Can I have/implement something
during normalization??


Thanks in advance


Regards,
Prateek