[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037358#comment-15037358 ] Modassar Ather edited comment on LUCENE-5205 at 12/3/15 6:41 AM: - Hi [~talli...@mitre.org] A prefix in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:term). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not be used the way it is. Thanks, Modassar was (Author: modassar): Hi [~talli...@mitre.org] A prefix in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:refasten). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not used the way it is. Thanks, Modassar > SpanQueryParser with recursion, analysis and syntax very similar to classic > QueryParser > --- > > Key: LUCENE-5205 > URL: https://issues.apache.org/jira/browse/LUCENE-5205 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Tim Allison > Labels: patch > Attachments: LUCENE-5205-cleanup-tests.patch, > LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, > LUCENE-5205_dateTestReInitPkgPrvt.patch, > LUCENE-5205_improve_stop_word_handling.patch, > LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, > SpanQueryParser_v1.patch.gz, patch.txt > > > This parser extends QueryParserBase and includes functionality from: > * Classic QueryParser: most of its syntax > * SurroundQueryParser: recursive parsing for "near" and "not" clauses. > * ComplexPhraseQueryParser: can handle "near" queries that include multiterms > (wildcard, fuzzy, regex, prefix), > * AnalyzingQueryParser: has an option to analyze multiterms. > At a high level, there's a first pass BooleanQuery/field parser and then a > span query parser handles all terminal nodes and phrases. > Same as classic syntax: > * term: test > * fuzzy: roam~0.8, roam~2 > * wildcard: te?t, test*, t*st > * regex: /\[mb\]oat/ > * phrase: "jakarta apache" > * phrase with slop: "jakarta apache"~3 > * default "or" clause: jakarta apache > * grouping "or" clause: (jakarta apache) > * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta > * multiple fields: title:lucene author:hatcher > > Main additions in SpanQueryParser syntax vs. classic syntax: > * Can require "in order" for phrases with slop with the \~> operator: > "jakarta apache"\~>3 > * Can specify "not near": "fever bieber"!\~3,10 :: > find "fever" but not if "bieber" appears within 3 words before or 10 > words after it. > * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta > apache\]~3 lucene\]\~>4 :: > find "jakarta" within 3 words of "apache", and that hit has to be within > four words before "lucene" > * Can also use \[\] for single level phrasal queries instead of " as in: > \[jakarta apache\] > * Can use "or grouping" clauses in phrasal queries: "apache (lucene solr)"\~3 > :: find "apache" and then either "lucene" or "solr" within three words. > * Can use multiterms in phrasal queries: "jakarta\~1 ap*che"\~2 > * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ > /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like "jakarta" within two > words of "ap*che" and that hit has to be within ten words of something like > "solr" or that "lucene" regex. > * Can require at least x number of hits at boolean level: "apache AND (lucene > solr tika)~2 > * Can use negative only query: -jakarta :: Find all docs that don't contain > "jakarta" > * Can use an edit distance > 2 for fuzzy query via SlowFuzzyQuery (beware of > potential performance issues!). > Trivial additions: > * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, > prefix =2) > * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance > <=2: (jakarta~1 (OSA) vs jakarta~>1(Levenshtein) > This parser can be very useful for concordance tasks (see also LUCENE-5317 > and LUCENE-5318) and for analytical search. > Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. > Most of the documentation is in the javadoc for SpanQueryParser. > Any and all feedback is welcome. Thank you. > Until this is added to the Lucene project, I've added a standalone > lucene-addons repo (with jars compiled for the latest stable build of Lucene) > on [github|https://github.com/tballison/lucene-addons]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037358#comment-15037358 ] Modassar Ather edited comment on LUCENE-5205 at 12/3/15 6:43 AM: - Hi [~talli...@mitre.org] Wildard (*) in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:term). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not be used the way it is in the example i.e a single wildcard term within double quotes. Thanks, Modassar was (Author: modassar): Hi [~talli...@mitre.org] A prefix in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:term). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not be used the way it is. Thanks, Modassar > SpanQueryParser with recursion, analysis and syntax very similar to classic > QueryParser > --- > > Key: LUCENE-5205 > URL: https://issues.apache.org/jira/browse/LUCENE-5205 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Tim Allison > Labels: patch > Attachments: LUCENE-5205-cleanup-tests.patch, > LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, > LUCENE-5205_dateTestReInitPkgPrvt.patch, > LUCENE-5205_improve_stop_word_handling.patch, > LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, > SpanQueryParser_v1.patch.gz, patch.txt > > > This parser extends QueryParserBase and includes functionality from: > * Classic QueryParser: most of its syntax > * SurroundQueryParser: recursive parsing for "near" and "not" clauses. > * ComplexPhraseQueryParser: can handle "near" queries that include multiterms > (wildcard, fuzzy, regex, prefix), > * AnalyzingQueryParser: has an option to analyze multiterms. > At a high level, there's a first pass BooleanQuery/field parser and then a > span query parser handles all terminal nodes and phrases. > Same as classic syntax: > * term: test > * fuzzy: roam~0.8, roam~2 > * wildcard: te?t, test*, t*st > * regex: /\[mb\]oat/ > * phrase: "jakarta apache" > * phrase with slop: "jakarta apache"~3 > * default "or" clause: jakarta apache > * grouping "or" clause: (jakarta apache) > * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta > * multiple fields: title:lucene author:hatcher > > Main additions in SpanQueryParser syntax vs. classic syntax: > * Can require "in order" for phrases with slop with the \~> operator: > "jakarta apache"\~>3 > * Can specify "not near": "fever bieber"!\~3,10 :: > find "fever" but not if "bieber" appears within 3 words before or 10 > words after it. > * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta > apache\]~3 lucene\]\~>4 :: > find "jakarta" within 3 words of "apache", and that hit has to be within > four words before "lucene" > * Can also use \[\] for single level phrasal queries instead of " as in: > \[jakarta apache\] > * Can use "or grouping" clauses in phrasal queries: "apache (lucene solr)"\~3 > :: find "apache" and then either "lucene" or "solr" within three words. > * Can use multiterms in phrasal queries: "jakarta\~1 ap*che"\~2 > * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ > /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like "jakarta" within two > words of "ap*che" and that hit has to be within ten words of something like > "solr" or that "lucene" regex. > * Can require at least x number of hits at boolean level: "apache AND (lucene > solr tika)~2 > * Can use negative only query: -jakarta :: Find all docs that don't contain > "jakarta" > * Can use an edit distance > 2 for fuzzy query via SlowFuzzyQuery (beware of > potential performance issues!). > Trivial additions: > * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, > prefix =2) > * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance > <=2: (jakarta~1 (OSA) vs jakarta~>1(Levenshtein) > This parser can be very useful for concordance tasks (see also LUCENE-5317 > and LUCENE-5318) and for analytical search. > Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. > Most of the documentation is in the javadoc for SpanQueryParser. > Any and all feedback is welcome. Thank you. > Until this is added to the Lucene project, I've added a standalone > lucene-addons repo (with jars compiled for the latest stable build of Lucene) > on [github|https://github.com/tballison/lucene-addons]. --
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037358#comment-15037358 ] Modassar Ather edited comment on LUCENE-5205 at 12/3/15 6:44 AM: - Hi [~talli...@mitre.org] Wildard * in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:term). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not be used the way it is in the example i.e a single wildcard term within double quotes. Thanks, Modassar was (Author: modassar): Hi [~talli...@mitre.org] Wildard (*) in double quotes is getting eaten up by the analyzer. E.g. f:"term*" is getting parsed to (+f:term). What I understand the wildcard queries are not analyzed. I noticed that the same works fine within a phrase. E.g. f:"term1 term*". Please let me know if it is an issue or it should not be used the way it is in the example i.e a single wildcard term within double quotes. Thanks, Modassar > SpanQueryParser with recursion, analysis and syntax very similar to classic > QueryParser > --- > > Key: LUCENE-5205 > URL: https://issues.apache.org/jira/browse/LUCENE-5205 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Tim Allison > Labels: patch > Attachments: LUCENE-5205-cleanup-tests.patch, > LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, > LUCENE-5205_dateTestReInitPkgPrvt.patch, > LUCENE-5205_improve_stop_word_handling.patch, > LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, > SpanQueryParser_v1.patch.gz, patch.txt > > > This parser extends QueryParserBase and includes functionality from: > * Classic QueryParser: most of its syntax > * SurroundQueryParser: recursive parsing for "near" and "not" clauses. > * ComplexPhraseQueryParser: can handle "near" queries that include multiterms > (wildcard, fuzzy, regex, prefix), > * AnalyzingQueryParser: has an option to analyze multiterms. > At a high level, there's a first pass BooleanQuery/field parser and then a > span query parser handles all terminal nodes and phrases. > Same as classic syntax: > * term: test > * fuzzy: roam~0.8, roam~2 > * wildcard: te?t, test*, t*st > * regex: /\[mb\]oat/ > * phrase: "jakarta apache" > * phrase with slop: "jakarta apache"~3 > * default "or" clause: jakarta apache > * grouping "or" clause: (jakarta apache) > * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta > * multiple fields: title:lucene author:hatcher > > Main additions in SpanQueryParser syntax vs. classic syntax: > * Can require "in order" for phrases with slop with the \~> operator: > "jakarta apache"\~>3 > * Can specify "not near": "fever bieber"!\~3,10 :: > find "fever" but not if "bieber" appears within 3 words before or 10 > words after it. > * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta > apache\]~3 lucene\]\~>4 :: > find "jakarta" within 3 words of "apache", and that hit has to be within > four words before "lucene" > * Can also use \[\] for single level phrasal queries instead of " as in: > \[jakarta apache\] > * Can use "or grouping" clauses in phrasal queries: "apache (lucene solr)"\~3 > :: find "apache" and then either "lucene" or "solr" within three words. > * Can use multiterms in phrasal queries: "jakarta\~1 ap*che"\~2 > * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ > /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like "jakarta" within two > words of "ap*che" and that hit has to be within ten words of something like > "solr" or that "lucene" regex. > * Can require at least x number of hits at boolean level: "apache AND (lucene > solr tika)~2 > * Can use negative only query: -jakarta :: Find all docs that don't contain > "jakarta" > * Can use an edit distance > 2 for fuzzy query via SlowFuzzyQuery (beware of > potential performance issues!). > Trivial additions: > * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, > prefix =2) > * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance > <=2: (jakarta~1 (OSA) vs jakarta~>1(Levenshtein) > This parser can be very useful for concordance tasks (see also LUCENE-5317 > and LUCENE-5318) and for analytical search. > Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. > Most of the documentation is in the javadoc for SpanQueryParser. > Any and all feedback is welcome. Thank you. > Until this is added to the Lucene project, I've added a standalone > lucene-addons repo (with jars compiled for the latest stable build of Lucene)
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033625#comment-15033625 ] Modassar Ather edited comment on LUCENE-5205 at 12/1/15 12:43 PM: -- Hi [~talli...@mitre.org] Kindly let me know how to escape special character in SpanQueryParser. From the discussion above it seems that single quote is used. I thinks I am missing some understanding about it as normally \ is used to escape special character. e.g "understanding (span query)" To escape brackets in above query what is the right character. I want to match the phrase where brackets are not considered as special character. Thanks, Modassar was (Author: modassar): Hi [~talli...@mitre.org] Kindly let me know how to escape special character in SpanQueryParser. From the discussion above it seems that single quote is used. I thinks I am missing some understanding about it as normally \ is used to escape special character. e.g "understanding (span query)" To escape brackets in above query what is the right character. I want a match the phrase where brackets are not considered as special character. Thanks, Modassar > SpanQueryParser with recursion, analysis and syntax very similar to classic > QueryParser > --- > > Key: LUCENE-5205 > URL: https://issues.apache.org/jira/browse/LUCENE-5205 > Project: Lucene - Core > Issue Type: Improvement > Components: core/queryparser >Reporter: Tim Allison > Labels: patch > Attachments: LUCENE-5205-cleanup-tests.patch, > LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, > LUCENE-5205_dateTestReInitPkgPrvt.patch, > LUCENE-5205_improve_stop_word_handling.patch, > LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, > SpanQueryParser_v1.patch.gz, patch.txt > > > This parser extends QueryParserBase and includes functionality from: > * Classic QueryParser: most of its syntax > * SurroundQueryParser: recursive parsing for "near" and "not" clauses. > * ComplexPhraseQueryParser: can handle "near" queries that include multiterms > (wildcard, fuzzy, regex, prefix), > * AnalyzingQueryParser: has an option to analyze multiterms. > At a high level, there's a first pass BooleanQuery/field parser and then a > span query parser handles all terminal nodes and phrases. > Same as classic syntax: > * term: test > * fuzzy: roam~0.8, roam~2 > * wildcard: te?t, test*, t*st > * regex: /\[mb\]oat/ > * phrase: "jakarta apache" > * phrase with slop: "jakarta apache"~3 > * default "or" clause: jakarta apache > * grouping "or" clause: (jakarta apache) > * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta > * multiple fields: title:lucene author:hatcher > > Main additions in SpanQueryParser syntax vs. classic syntax: > * Can require "in order" for phrases with slop with the \~> operator: > "jakarta apache"\~>3 > * Can specify "not near": "fever bieber"!\~3,10 :: > find "fever" but not if "bieber" appears within 3 words before or 10 > words after it. > * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta > apache\]~3 lucene\]\~>4 :: > find "jakarta" within 3 words of "apache", and that hit has to be within > four words before "lucene" > * Can also use \[\] for single level phrasal queries instead of " as in: > \[jakarta apache\] > * Can use "or grouping" clauses in phrasal queries: "apache (lucene solr)"\~3 > :: find "apache" and then either "lucene" or "solr" within three words. > * Can use multiterms in phrasal queries: "jakarta\~1 ap*che"\~2 > * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ > /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like "jakarta" within two > words of "ap*che" and that hit has to be within ten words of something like > "solr" or that "lucene" regex. > * Can require at least x number of hits at boolean level: "apache AND (lucene > solr tika)~2 > * Can use negative only query: -jakarta :: Find all docs that don't contain > "jakarta" > * Can use an edit distance > 2 for fuzzy query via SlowFuzzyQuery (beware of > potential performance issues!). > Trivial additions: > * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, > prefix =2) > * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance > <=2: (jakarta~1 (OSA) vs jakarta~>1(Levenshtein) > This parser can be very useful for concordance tasks (see also LUCENE-5317 > and LUCENE-5318) and for analytical search. > Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. > Most of the documentation is in the javadoc for SpanQueryParser. > Any and all feedback is welcome. Thank you. > Until this is added to the Lucene project, I've added a standalone > lucene-addons
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[
https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033952#comment-15033952
]
Tim Allison edited comment on LUCENE-5205 at 12/1/15 4:09 PM:
--
Let's say you are using a whitespace tokenizer and you're trying to find
{{(span}}, you'd surround that term with {{''}} as in {{'(span'}}.
To search for the phrase above:
{{"understanding '(span' 'query)'"}}
If your target token has a {{'}} in it, {{(sp'an}}, double the token's single
quotes: {{'(sp''an'}}
was (Author: talli...@mitre.org):
Let's say you are using a whitespace tokenizer and you're trying to find
{{(span}}, you'd surround that term with {{''}} as in {{'(span'}}.
To search for the phrase above:
{{"understanding '(span' 'query)'"}}
If your target token has a {{'}} in it, {{(sp'an}}, you'd use this:
{{'(sp''an'}}
> SpanQueryParser with recursion, analysis and syntax very similar to classic
> QueryParser
> ---
>
> Key: LUCENE-5205
> URL: https://issues.apache.org/jira/browse/LUCENE-5205
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/queryparser
>Reporter: Tim Allison
> Labels: patch
> Attachments: LUCENE-5205-cleanup-tests.patch,
> LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz,
> LUCENE-5205_dateTestReInitPkgPrvt.patch,
> LUCENE-5205_improve_stop_word_handling.patch,
> LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch,
> SpanQueryParser_v1.patch.gz, patch.txt
>
>
> This parser extends QueryParserBase and includes functionality from:
> * Classic QueryParser: most of its syntax
> * SurroundQueryParser: recursive parsing for "near" and "not" clauses.
> * ComplexPhraseQueryParser: can handle "near" queries that include multiterms
> (wildcard, fuzzy, regex, prefix),
> * AnalyzingQueryParser: has an option to analyze multiterms.
> At a high level, there's a first pass BooleanQuery/field parser and then a
> span query parser handles all terminal nodes and phrases.
> Same as classic syntax:
> * term: test
> * fuzzy: roam~0.8, roam~2
> * wildcard: te?t, test*, t*st
> * regex: /\[mb\]oat/
> * phrase: "jakarta apache"
> * phrase with slop: "jakarta apache"~3
> * default "or" clause: jakarta apache
> * grouping "or" clause: (jakarta apache)
> * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta
> * multiple fields: title:lucene author:hatcher
>
> Main additions in SpanQueryParser syntax vs. classic syntax:
> * Can require "in order" for phrases with slop with the \~> operator:
> "jakarta apache"\~>3
> * Can specify "not near": "fever bieber"!\~3,10 ::
> find "fever" but not if "bieber" appears within 3 words before or 10
> words after it.
> * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta
> apache\]~3 lucene\]\~>4 ::
> find "jakarta" within 3 words of "apache", and that hit has to be within
> four words before "lucene"
> * Can also use \[\] for single level phrasal queries instead of " as in:
> \[jakarta apache\]
> * Can use "or grouping" clauses in phrasal queries: "apache (lucene solr)"\~3
> :: find "apache" and then either "lucene" or "solr" within three words.
> * Can use multiterms in phrasal queries: "jakarta\~1 ap*che"\~2
> * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~
> /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like "jakarta" within two
> words of "ap*che" and that hit has to be within ten words of something like
> "solr" or that "lucene" regex.
> * Can require at least x number of hits at boolean level: "apache AND (lucene
> solr tika)~2
> * Can use negative only query: -jakarta :: Find all docs that don't contain
> "jakarta"
> * Can use an edit distance > 2 for fuzzy query via SlowFuzzyQuery (beware of
> potential performance issues!).
> Trivial additions:
> * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1,
> prefix =2)
> * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance
> <=2: (jakarta~1 (OSA) vs jakarta~>1(Levenshtein)
> This parser can be very useful for concordance tasks (see also LUCENE-5317
> and LUCENE-5318) and for analytical search.
> Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery.
> Most of the documentation is in the javadoc for SpanQueryParser.
> Any and all feedback is welcome. Thank you.
> Until this is added to the Lucene project, I've added a standalone
> lucene-addons repo (with jars compiled for the latest stable build of Lucene)
> on [github|https://github.com/tballison/lucene-addons].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[
https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734997#comment-14734997
]
Tim Allison edited comment on LUCENE-5205 at 9/8/15 3:34 PM:
-
This looks like a genuine issue in the Highlighter. I was hoping that it was
LUCENE-5503 so that would get some attention, but I don't think it is.
This is the minimal code to show the problem:
{code}
@Test
public void testEmbeddedSpanNearHighlighterIssue() throws Exception {
String field = "f";
Analyzer analyzer = new StandardAnalyzer();
String text = "b c d";
//SpanQueryParser p = new SpanQueryParser(field, analyzer);
//Query q = p.parse("\"(b [c z]) d\"~2");
SpanQuery cz = new SpanNearQuery(
new SpanQuery[]{
new SpanTermQuery(new Term(field, "c")),
new SpanTermQuery(new Term(field, "z"))
}, 0, true
);
SpanQuery bcz = new SpanOrQuery(
new SpanTermQuery(new Term(field, "b")),
cz);
SpanQuery q = new SpanNearQuery(
new SpanQuery[]{
bcz,
new SpanTermQuery(new Term(field, "d"))
}, 2, false
);
QueryScorer scorer = new QueryScorer(q, field);
scorer.setExpandMultiTermQuery(true);
Fragmenter fragmenter = new SimpleFragmenter(1000);
Highlighter highlighter = new Highlighter(
new SimpleHTMLFormatter(),
new SimpleHTMLEncoder(),
scorer);
highlighter.setTextFragmenter(fragmenter);
String[] snippets = highlighter.getBestFragments(analyzer,
field, text,
3);
assertEquals(1, snippets.length);
assertFalse(snippets[0].contains("c"));
}
{code}
This problem does not happen if "c" comes before "a" or after "d" in the text:
"c b d" or "b d c".
was (Author: talli...@mitre.org):
This looks like a genuine issue in the Highlighter. I was hoping that it was
LUCENE-5503 so that would get some attention, but I don't think it is.
This is the minimal code to show the problem:
{code}
@Test
public void testEmbeddedSpanNearHighlighterIssue() throws Exception {
String field = "f";
Analyzer analyzer = new StandardAnalyzer();
String text = "b c d";
//SpanQueryParser p = new SpanQueryParser("f", analyzer);
//Query q = p.parse("\"(b [c z]) d\"~2");
SpanQuery cz = new SpanNearQuery(
new SpanQuery[]{
new SpanTermQuery(new Term(field, "c")),
new SpanTermQuery(new Term(field, "z"))
}, 0, true
);
SpanQuery bcz = new SpanOrQuery(
new SpanTermQuery(new Term(field, "b")),
cz);
SpanQuery q = new SpanNearQuery(
new SpanQuery[]{
bcz,
new SpanTermQuery(new Term(field, "d"))
}, 2, false
);
QueryScorer scorer = new QueryScorer(q, "f");
scorer.setExpandMultiTermQuery(true);
Fragmenter fragmenter = new SimpleFragmenter(1000);
Highlighter highlighter = new Highlighter(
new SimpleHTMLFormatter(),
new SimpleHTMLEncoder(),
scorer);
highlighter.setTextFragmenter(fragmenter);
String[] snippets = highlighter.getBestFragments(analyzer,
"f", text,
3);
assertEquals(1, snippets.length);
assertFalse(snippets[0].contains("c"));
}
{code}
This problem does not happen if "c" comes before "a" or after "d" in the text:
"c b d" or "b d c".
> SpanQueryParser with recursion, analysis and syntax very similar to classic
> QueryParser
> ---
>
> Key: LUCENE-5205
> URL: https://issues.apache.org/jira/browse/LUCENE-5205
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/queryparser
>Reporter: Tim Allison
> Labels: patch
> Attachments: LUCENE-5205-cleanup-tests.patch,
> LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz,
> LUCENE-5205_dateTestReInitPkgPrvt.patch,
> LUCENE-5205_improve_stop_word_handling.patch,
> LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch,
> SpanQueryParser_v1.patch.gz, patch.txt
>
>
> This parser extends QueryParserBase and includes functionality from:
> * Classic QueryParser: most of its syntax
> * SurroundQueryParser: recursive parsing for "near" and "not" clauses.
> * ComplexPhraseQueryParser: can handle "near" queries that include multiterms
> (wildcard, fuzzy, regex, prefix),
> * AnalyzingQueryParser: has an option to analyze multiterms.
> At a high level, there's a first pass BooleanQuery/field parser and then a
> span query parser handles all terminal nodes and phrases.
> Same as classic syntax:
> * term: test
> * fuzzy: roam~0.8, roam~2
> * wildcard: te?t, test*, t*st
> * regex: /\[mb\]oat/
> * phrase: "jakarta apache"
> * phrase with slop: "jakarta apache"~3
> * default "or" clause: jakarta apache
>
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596593#comment-14596593 ] Tim Allison edited comment on LUCENE-5205 at 6/22/15 8:29 PM: -- Cleaned up new lexer in lexer2 branch. Pre-release compiled jars are available [here|https://github.com/tballison/lucene-addons/releases/tag/5x-1.1] was (Author: talli...@mitre.org): Cleaned up new lexer in lexer2 branch. Prelease compiled jars are available [here|https://github.com/tballison/lucene-addons/releases/tag/5x-1.1] SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser --- Key: LUCENE-5205 URL: https://issues.apache.org/jira/browse/LUCENE-5205 Project: Lucene - Core Issue Type: Improvement Components: core/queryparser Reporter: Tim Allison Labels: patch Attachments: LUCENE-5205-cleanup-tests.patch, LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, LUCENE-5205_dateTestReInitPkgPrvt.patch, LUCENE-5205_improve_stop_word_handling.patch, LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, SpanQueryParser_v1.patch.gz, patch.txt This parser extends QueryParserBase and includes functionality from: * Classic QueryParser: most of its syntax * SurroundQueryParser: recursive parsing for near and not clauses. * ComplexPhraseQueryParser: can handle near queries that include multiterms (wildcard, fuzzy, regex, prefix), * AnalyzingQueryParser: has an option to analyze multiterms. At a high level, there's a first pass BooleanQuery/field parser and then a span query parser handles all terminal nodes and phrases. Same as classic syntax: * term: test * fuzzy: roam~0.8, roam~2 * wildcard: te?t, test*, t*st * regex: /\[mb\]oat/ * phrase: jakarta apache * phrase with slop: jakarta apache~3 * default or clause: jakarta apache * grouping or clause: (jakarta apache) * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta * multiple fields: title:lucene author:hatcher Main additions in SpanQueryParser syntax vs. classic syntax: * Can require in order for phrases with slop with the \~ operator: jakarta apache\~3 * Can specify not near: fever bieber!\~3,10 :: find fever but not if bieber appears within 3 words before or 10 words after it. * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta apache\]~3 lucene\]\~4 :: find jakarta within 3 words of apache, and that hit has to be within four words before lucene * Can also use \[\] for single level phrasal queries instead of as in: \[jakarta apache\] * Can use or grouping clauses in phrasal queries: apache (lucene solr)\~3 :: find apache and then either lucene or solr within three words. * Can use multiterms in phrasal queries: jakarta\~1 ap*che\~2 * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like jakarta within two words of ap*che and that hit has to be within ten words of something like solr or that lucene regex. * Can require at least x number of hits at boolean level: apache AND (lucene solr tika)~2 * Can use negative only query: -jakarta :: Find all docs that don't contain jakarta * Can use an edit distance 2 for fuzzy query via SlowFuzzyQuery (beware of potential performance issues!). Trivial additions: * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, prefix =2) * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance =2: (jakarta~1 (OSA) vs jakarta~1(Levenshtein) This parser can be very useful for concordance tasks (see also LUCENE-5317 and LUCENE-5318) and for analytical search. Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. Most of the documentation is in the javadoc for SpanQueryParser. Any and all feedback is welcome. Thank you. Until this is added to the Lucene project, I've added a standalone lucene-addons repo (with jars compiled for the latest stable build of Lucene) on [github|https://github.com/tballison/lucene-addons]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595763#comment-14595763 ] Tim Allison edited comment on LUCENE-5205 at 6/22/15 11:04 AM: --- Fixed in [solr-collab-5x branch|https://github.com/tballison/lucene-addons/tree/solr-collab-5x]. Over the next week, I hope to finish the implementation of the new lexer (get rid of the ugly regex) and do some general clean up. [~modassar], thank you for reporting this. Let me know if there are any other surprises. Oh, and on your most recent post about the stopwords, y, that caused at least one unit test to blow up with 5.2.1. Thank you, again. was (Author: talli...@mitre.org): Fixed in [solr-collab-5x branch|https://github.com/tballison/lucene-addons/tree/solr-collab-5x]. Over the next week, I hope to finish the implementation of the new lexer (get rid of the ugly regex) and do some general clean up. [~modassar], thank you for reporting this. Let me know if there are any other surprises. SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser --- Key: LUCENE-5205 URL: https://issues.apache.org/jira/browse/LUCENE-5205 Project: Lucene - Core Issue Type: Improvement Components: core/queryparser Reporter: Tim Allison Labels: patch Attachments: LUCENE-5205-cleanup-tests.patch, LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, LUCENE-5205_dateTestReInitPkgPrvt.patch, LUCENE-5205_improve_stop_word_handling.patch, LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, SpanQueryParser_v1.patch.gz, patch.txt This parser extends QueryParserBase and includes functionality from: * Classic QueryParser: most of its syntax * SurroundQueryParser: recursive parsing for near and not clauses. * ComplexPhraseQueryParser: can handle near queries that include multiterms (wildcard, fuzzy, regex, prefix), * AnalyzingQueryParser: has an option to analyze multiterms. At a high level, there's a first pass BooleanQuery/field parser and then a span query parser handles all terminal nodes and phrases. Same as classic syntax: * term: test * fuzzy: roam~0.8, roam~2 * wildcard: te?t, test*, t*st * regex: /\[mb\]oat/ * phrase: jakarta apache * phrase with slop: jakarta apache~3 * default or clause: jakarta apache * grouping or clause: (jakarta apache) * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta * multiple fields: title:lucene author:hatcher Main additions in SpanQueryParser syntax vs. classic syntax: * Can require in order for phrases with slop with the \~ operator: jakarta apache\~3 * Can specify not near: fever bieber!\~3,10 :: find fever but not if bieber appears within 3 words before or 10 words after it. * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta apache\]~3 lucene\]\~4 :: find jakarta within 3 words of apache, and that hit has to be within four words before lucene * Can also use \[\] for single level phrasal queries instead of as in: \[jakarta apache\] * Can use or grouping clauses in phrasal queries: apache (lucene solr)\~3 :: find apache and then either lucene or solr within three words. * Can use multiterms in phrasal queries: jakarta\~1 ap*che\~2 * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like jakarta within two words of ap*che and that hit has to be within ten words of something like solr or that lucene regex. * Can require at least x number of hits at boolean level: apache AND (lucene solr tika)~2 * Can use negative only query: -jakarta :: Find all docs that don't contain jakarta * Can use an edit distance 2 for fuzzy query via SlowFuzzyQuery (beware of potential performance issues!). Trivial additions: * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, prefix =2) * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance =2: (jakarta~1 (OSA) vs jakarta~1(Levenshtein) This parser can be very useful for concordance tasks (see also LUCENE-5317 and LUCENE-5318) and for analytical search. Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. Most of the documentation is in the javadoc for SpanQueryParser. Any and all feedback is welcome. Thank you. Until this is added to the Lucene project, I've added a standalone lucene-addons repo (with jars compiled for the latest stable build of Lucene) on [github|https://github.com/tballison/lucene-addons]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[
https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591616#comment-14591616
]
Modassar Ather edited comment on LUCENE-5205 at 6/18/15 10:43 AM:
--
Adding one more query for better clarification.
{noformat}[(term1 [term2 term3]~1 [term4 term5]~1) (term6 [term7 term8]~1
term9)]~2 (term10 [term11 term12])~4{noformat}
was (Author: modassar):
Adding one more query for better clarification.
{noformat}[(term1 [term2 term3]~1 [term term]~1) (term4 [term5 term6]~1
term7)]~2 (term8 [term9 term10])~4{noformat}
SpanQueryParser with recursion, analysis and syntax very similar to classic
QueryParser
---
Key: LUCENE-5205
URL: https://issues.apache.org/jira/browse/LUCENE-5205
Project: Lucene - Core
Issue Type: Improvement
Components: core/queryparser
Reporter: Tim Allison
Labels: patch
Attachments: LUCENE-5205-cleanup-tests.patch,
LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz,
LUCENE-5205_dateTestReInitPkgPrvt.patch,
LUCENE-5205_improve_stop_word_handling.patch,
LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch,
SpanQueryParser_v1.patch.gz, patch.txt
This parser extends QueryParserBase and includes functionality from:
* Classic QueryParser: most of its syntax
* SurroundQueryParser: recursive parsing for near and not clauses.
* ComplexPhraseQueryParser: can handle near queries that include multiterms
(wildcard, fuzzy, regex, prefix),
* AnalyzingQueryParser: has an option to analyze multiterms.
At a high level, there's a first pass BooleanQuery/field parser and then a
span query parser handles all terminal nodes and phrases.
Same as classic syntax:
* term: test
* fuzzy: roam~0.8, roam~2
* wildcard: te?t, test*, t*st
* regex: /\[mb\]oat/
* phrase: jakarta apache
* phrase with slop: jakarta apache~3
* default or clause: jakarta apache
* grouping or clause: (jakarta apache)
* boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta
* multiple fields: title:lucene author:hatcher
Main additions in SpanQueryParser syntax vs. classic syntax:
* Can require in order for phrases with slop with the \~ operator:
jakarta apache\~3
* Can specify not near: fever bieber!\~3,10 ::
find fever but not if bieber appears within 3 words before or 10
words after it.
* Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta
apache\]~3 lucene\]\~4 ::
find jakarta within 3 words of apache, and that hit has to be within
four words before lucene
* Can also use \[\] for single level phrasal queries instead of as in:
\[jakarta apache\]
* Can use or grouping clauses in phrasal queries: apache (lucene solr)\~3
:: find apache and then either lucene or solr within three words.
* Can use multiterms in phrasal queries: jakarta\~1 ap*che\~2
* Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~
/l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like jakarta within two
words of ap*che and that hit has to be within ten words of something like
solr or that lucene regex.
* Can require at least x number of hits at boolean level: apache AND (lucene
solr tika)~2
* Can use negative only query: -jakarta :: Find all docs that don't contain
jakarta
* Can use an edit distance 2 for fuzzy query via SlowFuzzyQuery (beware of
potential performance issues!).
Trivial additions:
* Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1,
prefix =2)
* Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance
=2: (jakarta~1 (OSA) vs jakarta~1(Levenshtein)
This parser can be very useful for concordance tasks (see also LUCENE-5317
and LUCENE-5318) and for analytical search.
Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery.
Most of the documentation is in the javadoc for SpanQueryParser.
Any and all feedback is welcome. Thank you.
Until this is added to the Lucene project, I've added a standalone
lucene-addons repo (with jars compiled for the latest stable build of Lucene)
on [github|https://github.com/tballison/lucene-addons].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-5205) SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser
[ https://issues.apache.org/jira/browse/LUCENE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231476#comment-14231476 ] Tim Allison edited comment on LUCENE-5205 at 12/2/14 1:39 PM: -- Will look into it. To confirm, you have a query with 6000-8000 terms that you are OR'ing together? [~dsmiley], do I remember correctly that you recently added a query type for efficiently searching lots of terms? was (Author: talli...@mitre.org): Will look into it. To confirm, you have a query with 6000-8000 terms that you are OR'ing together? [~dsmiley], do I remember correctly that you recently added a query type for lots of terms? SpanQueryParser with recursion, analysis and syntax very similar to classic QueryParser --- Key: LUCENE-5205 URL: https://issues.apache.org/jira/browse/LUCENE-5205 Project: Lucene - Core Issue Type: Improvement Components: core/queryparser Reporter: Tim Allison Labels: patch Attachments: LUCENE-5205-cleanup-tests.patch, LUCENE-5205-date-pkg-prvt.patch, LUCENE-5205.patch.gz, LUCENE-5205.patch.gz, LUCENE-5205_dateTestReInitPkgPrvt.patch, LUCENE-5205_improve_stop_word_handling.patch, LUCENE-5205_smallTestMods.patch, LUCENE_5205.patch, SpanQueryParser_v1.patch.gz, patch.txt This parser extends QueryParserBase and includes functionality from: * Classic QueryParser: most of its syntax * SurroundQueryParser: recursive parsing for near and not clauses. * ComplexPhraseQueryParser: can handle near queries that include multiterms (wildcard, fuzzy, regex, prefix), * AnalyzingQueryParser: has an option to analyze multiterms. At a high level, there's a first pass BooleanQuery/field parser and then a span query parser handles all terminal nodes and phrases. Same as classic syntax: * term: test * fuzzy: roam~0.8, roam~2 * wildcard: te?t, test*, t*st * regex: /\[mb\]oat/ * phrase: jakarta apache * phrase with slop: jakarta apache~3 * default or clause: jakarta apache * grouping or clause: (jakarta apache) * boolean and +/-: (lucene OR apache) NOT jakarta; +lucene +apache -jakarta * multiple fields: title:lucene author:hatcher Main additions in SpanQueryParser syntax vs. classic syntax: * Can require in order for phrases with slop with the \~ operator: jakarta apache\~3 * Can specify not near: fever bieber!\~3,10 :: find fever but not if bieber appears within 3 words before or 10 words after it. * Fully recursive phrasal queries with \[ and \]; as in: \[\[jakarta apache\]~3 lucene\]\~4 :: find jakarta within 3 words of apache, and that hit has to be within four words before lucene * Can also use \[\] for single level phrasal queries instead of as in: \[jakarta apache\] * Can use or grouping clauses in phrasal queries: apache (lucene solr)\~3 :: find apache and then either lucene or solr within three words. * Can use multiterms in phrasal queries: jakarta\~1 ap*che\~2 * Did I mention full recursion: \[\[jakarta\~1 ap*che\]\~2 (solr~ /l\[ou\]\+\[cs\]\[en\]\+/)]\~10 :: Find something like jakarta within two words of ap*che and that hit has to be within ten words of something like solr or that lucene regex. * Can require at least x number of hits at boolean level: apache AND (lucene solr tika)~2 * Can use negative only query: -jakarta :: Find all docs that don't contain jakarta * Can use an edit distance 2 for fuzzy query via SlowFuzzyQuery (beware of potential performance issues!). Trivial additions: * Can specify prefix length in fuzzy queries: jakarta~1,2 (edit distance =1, prefix =2) * Can specifiy Optimal String Alignment (OSA) vs Levenshtein for distance =2: (jakarta~1 (OSA) vs jakarta~1(Levenshtein) This parser can be very useful for concordance tasks (see also LUCENE-5317 and LUCENE-5318) and for analytical search. Until LUCENE-2878 is closed, this might have a use for fans of SpanQuery. Most of the documentation is in the javadoc for SpanQueryParser. Any and all feedback is welcome. Thank you. Until this is added to the Lucene project, I've added a standalone lucene-addons repo (with jars compiled for the latest stable build of Lucene) on [github|https://github.com/tballison/lucene-addons]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
