Re: [HACKERS] 9.6 phrase search distance specification
On Thu, Aug 11, 2016 at 10:42 AM, Ryan Pedela wrote: > On Thu, Aug 11, 2016 at 9:27 AM, Oleg Bartunov > wrote: > >> On Tue, Aug 9, 2016 at 9:59 PM, Ryan Pedela >> wrote: >> > >> > >> >> > I would say that it is worth it to have a "phrase slop" operator >> (Apache >> > Lucene terminology). Proximity search is extremely useful for improving >> > relevance and phrase slop is one of the tools to achieve that. >> > >> >> It'd be great if you explain what is "phrase slop". I assume it's not >> about search, but about relevance. >> > > Sure. An exact phrase query has slop = 0 which means find all terms in the > exact positions relative to each other. Phrase query with slop > 0 means > find all terms within positions relative to each other. If slop = > 10, find all terms within 10 positions of each other. Here is a concrete > example from my current work searching SEC filings. > > Bill Gates' full legal name is William H. Gates, III. In the SEC database > [1], his name is GATES WILLIAM H III. If you are searching the records of > people within the SEC database and you want to find Bill Gates, most users > will type "bill gates". Since there are many people with the first name > Bill (William) and the last name Gates, Bill Gates most likely won't be the > first result with a standard keyword query. Likewise an exact phrase query > (slop = 0) will not find him either because the first and last names are > transposed. What you need is a phrase query with a slop = 2 which will > match "William Gates", "William H Gates", "Gates William", etc. There is > still the issue of Bill vs William, but that can be solved with synonyms > and is a different topic. > > 1. https://www.sec.gov/cgi-bin/browse-edgar?CIK=902012&owner > =exclude&action=getcompany&Find=Search > One more thing. In that trivial example, an AND query would probably do a great job too. However if you are searching for Bill Gates in large text documents rather than a list of names, an AND query will not give you very good results because the words "bill" and "gates" are so common.
Re: [HACKERS] 9.6 phrase search distance specification
On Thu, Aug 11, 2016 at 9:27 AM, Oleg Bartunov wrote: > On Tue, Aug 9, 2016 at 9:59 PM, Ryan Pedela > wrote: > > > > > > > I would say that it is worth it to have a "phrase slop" operator (Apache > > Lucene terminology). Proximity search is extremely useful for improving > > relevance and phrase slop is one of the tools to achieve that. > > > > It'd be great if you explain what is "phrase slop". I assume it's not > about search, but about relevance. > Sure. An exact phrase query has slop = 0 which means find all terms in the exact positions relative to each other. Phrase query with slop > 0 means find all terms within positions relative to each other. If slop = 10, find all terms within 10 positions of each other. Here is a concrete example from my current work searching SEC filings. Bill Gates' full legal name is William H. Gates, III. In the SEC database [1], his name is GATES WILLIAM H III. If you are searching the records of people within the SEC database and you want to find Bill Gates, most users will type "bill gates". Since there are many people with the first name Bill (William) and the last name Gates, Bill Gates most likely won't be the first result with a standard keyword query. Likewise an exact phrase query (slop = 0) will not find him either because the first and last names are transposed. What you need is a phrase query with a slop = 2 which will match "William Gates", "William H Gates", "Gates William", etc. There is still the issue of Bill vs William, but that can be solved with synonyms and is a different topic. 1. https://www.sec.gov/cgi-bin/browse-edgar?CIK=902012&owner=exclude&action= getcompany&Find=Search Thanks, Ryan
Re: [HACKERS] 9.6 phrase search distance specification
On Tue, Aug 9, 2016 at 9:59 PM, Ryan Pedela wrote: > > > I would say that it is worth it to have a "phrase slop" operator (Apache > Lucene terminology). Proximity search is extremely useful for improving > relevance and phrase slop is one of the tools to achieve that. > It'd be great if you explain what is "phrase slop". I assume it's not about search, but about relevance. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.6 phrase search distance specification
On Tue, Aug 9, 2016 at 12:59 PM, Ryan Pedela wrote: > > > Thanks, > > Ryan Pedela > Datalanche CEO, founder > www.datalanche.com > > On Tue, Aug 9, 2016 at 11:58 AM, Tom Lane wrote: > >> Bruce Momjian writes: >> > Does anyone know why the phrase distance "<3>" was changed from "at most >> > three tokens away" to "exactly three tokens away"? >> >> So that it would correctly support phraseto_tsquery's use of the operator >> to represent omitted words (stopwords) in a phrase. >> >> I think there's probably some use in also providing an operator that does >> "at most this many tokens away", but Oleg/Teodor were evidently less >> excited, because they didn't take the time to do it. >> >> The thread where this change was discussed is >> >> https://www.postgresql.org/message-id/flat/c19fcfec308e6ccd9 >> 52cdde9e648b505%40mail.gmail.com >> >> see particularly >> >> https://www.postgresql.org/message-id/11252.1465422251%40sss.pgh.pa.us > > > I would say that it is worth it to have a "phrase slop" operator (Apache > Lucene terminology). Proximity search is extremely useful for improving > relevance and phrase slop is one of the tools to achieve that. > > Sorry for the position of my signature Ryan
Re: [HACKERS] 9.6 phrase search distance specification
Thanks, Ryan Pedela Datalanche CEO, founder www.datalanche.com On Tue, Aug 9, 2016 at 11:58 AM, Tom Lane wrote: > Bruce Momjian writes: > > Does anyone know why the phrase distance "<3>" was changed from "at most > > three tokens away" to "exactly three tokens away"? > > So that it would correctly support phraseto_tsquery's use of the operator > to represent omitted words (stopwords) in a phrase. > > I think there's probably some use in also providing an operator that does > "at most this many tokens away", but Oleg/Teodor were evidently less > excited, because they didn't take the time to do it. > > The thread where this change was discussed is > > https://www.postgresql.org/message-id/flat/c19fcfec308e6ccd952cdde9e648b5 > 05%40mail.gmail.com > > see particularly > > https://www.postgresql.org/message-id/11252.1465422251%40sss.pgh.pa.us I would say that it is worth it to have a "phrase slop" operator (Apache Lucene terminology). Proximity search is extremely useful for improving relevance and phrase slop is one of the tools to achieve that.
Re: [HACKERS] 9.6 phrase search distance specification
On Tue, Aug 9, 2016 at 01:58:25PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > Does anyone know why the phrase distance "<3>" was changed from "at most > > three tokens away" to "exactly three tokens away"? > > So that it would correctly support phraseto_tsquery's use of the operator > to represent omitted words (stopwords) in a phrase. > > I think there's probably some use in also providing an operator that does > "at most this many tokens away", but Oleg/Teodor were evidently less > excited, because they didn't take the time to do it. > > The thread where this change was discussed is > > https://www.postgresql.org/message-id/flat/c19fcfec308e6ccd952cdde9e648b505%40mail.gmail.com > > see particularly > > https://www.postgresql.org/message-id/11252.1465422251%40sss.pgh.pa.us Ah, I know it was discussed somewhere. Thanks, the phraseto_tsquery tie-in was what I forgot. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.6 phrase search distance specification
Bruce Momjian writes: > Does anyone know why the phrase distance "<3>" was changed from "at most > three tokens away" to "exactly three tokens away"? So that it would correctly support phraseto_tsquery's use of the operator to represent omitted words (stopwords) in a phrase. I think there's probably some use in also providing an operator that does "at most this many tokens away", but Oleg/Teodor were evidently less excited, because they didn't take the time to do it. The thread where this change was discussed is https://www.postgresql.org/message-id/flat/c19fcfec308e6ccd952cdde9e648b505%40mail.gmail.com see particularly https://www.postgresql.org/message-id/11252.1465422251%40sss.pgh.pa.us regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] 9.6 phrase search distance specification
Does anyone know why the phrase distance "<3>" was changed from "at most three tokens away" to "exactly three tokens away"? I looked at the thread at: https://www.postgresql.org/message-id/flat/33828354.WrrSMviC7Y%40abook and didn't see the answer. I assume if you are looking for "<3>" you would want "<2>" matches and "<1>" matches as well. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
