A variant on this particular request: I would like to get the tf-idf for an indexed field. (the field is a body of a news document). I would like to find discriminating terms in the document set (the document set is a result of executing a filter on the search index. The discriminating terms are to help with improving the query as the number of documents returned are too many and relevant documents are getting lost in the search result (of executing a filter).
Is it possible to run the tf-idf calculations that Elastic does while indexing the document.(the API to access the TF-IDF calculations) Thanks Ramdev On Thursday, 17 October 2013 04:12:35 UTC-5, Jörg Prante wrote: > > I love to help people getting in touch with Elasticsearch. Don't hesitate > to ask questions - there are no silly questions, only silly answers. > > Thanks and kudos to Shay who is so dedicated and passionate and great by > deciding to release so valuable distributed scalable code to the public! > > Jörg > Am 17.10.2013 07:23 schrieb "Ivan Brusic" <[email protected] <javascript:> > >: > >> Thank Jorg for the plugin. The code is standard in Lucene (accessing the >> TermEnums), just adapted for elasticsearch. >> >> -- >> Ivan >> >> >> On Wed, Oct 16, 2013 at 5:30 PM, Ben McCann <[email protected]<javascript:> >> > wrote: >> >>> Thanks Ivan! Jorg's plugin is exactly what I was looking for. I just >>> sent him a pull request to update to 0.90.5 and he released a new version >>> with it, so I'll give it a go and see how it works. Also, good point that >>> tf-idf might not be exactly the right term for what I was looking for. I >>> mainly care about the IDF portion. >>> >>> Thanks again! >>> >>> -Ben >>> >>> P.S. you've helped me a couple times, so just wanted to say thanks! And >>> also I'm sure you get just as much recruiter spam as I do, so i won't bug >>> you, but if you ever want to explore the possibility of working on >>> elasticsearch with a bunch of ex-googlers then I'd love to share with you >>> what we're up to in case it's interesting to you >>> >>> >>> >>> >>> On Wed, Oct 16, 2013 at 12:57 PM, Ivan Brusic <[email protected]<javascript:> >>> > wrote: >>> >>>> Not quite the explain API, but the score explanation for any query: >>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-explain.html >>>> >>>> When enabled, it is similar to the Explain API, but for each document >>>> returned by the query instead of just one. >>>> >>>> TF-IDF only matters in the context of a query. If you want all the >>>> terms, you can use a term facet with a large size, or use Jorg's plugin: >>>> https://github.com/jprante/elasticsearch-index-termlist >>>> >>>> Cheers, >>>> >>>> Ivan >>>> >>>> >>>> >>>> >>>> On Wed, Oct 16, 2013 at 12:38 PM, Ben McCann >>>> <[email protected]<javascript:> >>>> > wrote: >>>> >>>>> Hi Ivan, >>>>> >>>>> Thanks for the tip! I'm not familiar with the explanation. Is that >>>>> the same as the Explain >>>>> API<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-explain.html> >>>>> for computing >>>>> a score explanation for a query and a specific document? I'd really like >>>>> to >>>>> get a list of all the terms that appear in a field in my index and not >>>>> only >>>>> for a particular query. >>>>> >>>>> Thanks, >>>>> Ben >>>>> >>>>> >>>>> >>>>> On Wed, Oct 16, 2013 at 12:27 PM, Ivan Brusic >>>>> <[email protected]<javascript:> >>>>> > wrote: >>>>> >>>>>> The various elements of scoring are exposed in the explanation (if >>>>>> enabled). Not an ideal format to process programmatically, but the >>>>>> results >>>>>> are there. >>>>>> >>>>>> TF-IDF is calculated per-field, with the score of the document being >>>>>> a combination of the various TF-IDF of the fields involved. >>>>>> >>>>>> -- >>>>>> Ivan >>>>>> >>>>>> >>>>>> On Wed, Oct 16, 2013 at 11:34 AM, Ben McCann >>>>>> <[email protected]<javascript:> >>>>>> > wrote: >>>>>> >>>>>>> Can you access the tf-idf to use outside of ElasticSearch? Also, is >>>>>>> the tf-idf calculated on a per-field basis or a per-document basis? >>>>>>> >>>>>>> Thanks, >>>>>>> Ben >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected] <javascript:>. >>>>>>> >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "elasticsearch" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/elasticsearch/ZhDlIksA5pE/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected] <javascript:>. >>>>>> >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> about.me/benmccann >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected] <javascript:>. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/ZhDlIksA5pE/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] <javascript:>. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> >>> >>> -- >>> about.me/benmccann >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd6c72a7-9f3f-4764-b512-9234a6769958%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
