Re: maxClauseCount Exception
bq: So all I want to do is a simple all docs with something in this field,
and to highlight the field
But that doesn't really make sense to do at the Solr/Lucene level. All
you're saying is that you want that field highlighted. Wouldn't it be much
easier to just do this at the app level whenever your field had anything
returned in it?
Best
Erick
On Sat, Mar 17, 2012 at 8:07 PM, Darren Govoni dar...@ontrenet.com wrote:
Thanks for the tip Hoss.
I notice that it appears sometimes and was varying because my index runs
would sometimes have different amount of docs, etc.
So all I want to do is a simple all docs with something in this field,
and to highlight the field.
Is the query expansion to all possible terms in the index really
necessary? I could have 100's of thousands of possible terms. Why should
they all become explicit query elements? Seems overkill and
underperformant.
Is there a another way with Lucene or not really?
On Thu, 2012-03-08 at 16:18 -0800, Chris Hostetter wrote:
: I am suddenly getting a maxClauseCount exception for no reason. I am
: using Solr 3.5. I have only 206 documents in my index.
Unless things have changed the reason you are seeing this is because
_highlighting_ a query (clause) like type_s:[*+TO+*] requires rewriting
it into a giant boolean query of all the terms in that field -- so even if
you only have 206 docs, if you have more then 206 values in that field in
your index, you're going to go over 1024 terms.
(you don't get this problem in a basic query, because it doens't need to
enumerate all the terms, it rewrites it to a ConstantScoreQuery)
what you most likeley want to do, is move some of those clauses like
type_s:[*+TO+*]: and usergroup_sm:admin) out of your main q query and
into fq filters ... so they can be cached independently, won't
contribute to scoring (just matching) and won't be used in highlighting.
:
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
: [#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
: org.apache.solr.servlet.SolrDispatchFilter|
: _ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
: $TooManyClauses: maxClauseCount is set to 1024
: at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
...
: at
: org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
: at
:
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
-Hoss
RE: Re: maxClauseCount Exception
true. but how can you find documents containing that field without expanding
1000 clauses?
brbrbr--- Original Message ---
On 3/19/2012 07:24 AM Erick Erickson wrote:brbq: So all I want to do is a simple
all docs with something in this field,
brand to highlight the field
br
brBut that doesn't really make sense to do at the Solr/Lucene level. All
bryou're saying is that you want that field highlighted. Wouldn't it be much
breasier to just do this at the app level whenever your field had anything
brreturned in it?
br
brBest
brErick
br
brOn Sat, Mar 17, 2012 at 8:07 PM, Darren Govoni dar...@ontrenet.com wrote:
br Thanks for the tip Hoss.
br
br I notice that it appears sometimes and was varying because my index runs
br would sometimes have different amount of docs, etc.
br
br So all I want to do is a simple all docs with something in this field,
br and to highlight the field.
br
br Is the query expansion to all possible terms in the index really
br necessary? I could have 100's of thousands of possible terms. Why should
br they all become explicit query elements? Seems overkill and
br underperformant.
br
br Is there a another way with Lucene or not really?
br
br On Thu, 2012-03-08 at 16:18 -0800, Chris Hostetter wrote:
br : I am suddenly getting a maxClauseCount exception for no reason. I am
br : using Solr 3.5. I have only 206 documents in my index.
br
br Unless things have changed the reason you are seeing this is because
br _highlighting_ a query (clause) like type_s:[*+TO+*] requires rewriting
br it into a giant boolean query of all the terms in that field -- so even
if
br you only have 206 docs, if you have more then 206 values in that field in
br your index, you're going to go over 1024 terms.
br
br (you don't get this problem in a basic query, because it doens't need to
br enumerate all the terms, it rewrites it to a ConstantScoreQuery)
br
br what you most likeley want to do, is move some of those clauses like
br type_s:[*+TO+*]: and usergroup_sm:admin) out of your main q query
and
br into fq filters ... so they can be cached independently, won't
br contribute to scoring (just matching) and won't be used in highlighting.
br
br :
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
br
br : [#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
br : org.apache.solr.servlet.SolrDispatchFilter|
br :
_ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
br : $TooManyClauses: maxClauseCount is set to 1024
br : at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
br ...
br : at
br :
org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
br : at
br :
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
br
br -Hoss
br
br
br
br
br
Re: Re: maxClauseCount Exception
Don't highlight on it. As Chris pointed out, it's the_highlighting_
that builds this ginormous clause. Just something like _searching_
on text:* has short-circuits in the code that refrain from this
expansion.
Best
Erick
On Mon, Mar 19, 2012 at 8:42 AM, Darren Govoni ontre...@ontrenet.com wrote:
true. but how can you find documents containing that field without expanding
1000 clauses?
brbrbr--- Original Message ---
On 3/19/2012 07:24 AM Erick Erickson wrote:brbq: So all I want to do is a
simple all docs with something in this field,
brand to highlight the field
br
brBut that doesn't really make sense to do at the Solr/Lucene level. All
bryou're saying is that you want that field highlighted. Wouldn't it be
much
breasier to just do this at the app level whenever your field had anything
brreturned in it?
br
brBest
brErick
br
brOn Sat, Mar 17, 2012 at 8:07 PM, Darren Govoni dar...@ontrenet.com
wrote:
br Thanks for the tip Hoss.
br
br I notice that it appears sometimes and was varying because my index
runs
br would sometimes have different amount of docs, etc.
br
br So all I want to do is a simple all docs with something in this
field,
br and to highlight the field.
br
br Is the query expansion to all possible terms in the index really
br necessary? I could have 100's of thousands of possible terms. Why
should
br they all become explicit query elements? Seems overkill and
br underperformant.
br
br Is there a another way with Lucene or not really?
br
br On Thu, 2012-03-08 at 16:18 -0800, Chris Hostetter wrote:
br : I am suddenly getting a maxClauseCount exception for no reason. I
am
br : using Solr 3.5. I have only 206 documents in my index.
br
br Unless things have changed the reason you are seeing this is because
br _highlighting_ a query (clause) like type_s:[*+TO+*] requires
rewriting
br it into a giant boolean query of all the terms in that field -- so
even if
br you only have 206 docs, if you have more then 206 values in that
field in
br your index, you're going to go over 1024 terms.
br
br (you don't get this problem in a basic query, because it doens't need
to
br enumerate all the terms, it rewrites it to a ConstantScoreQuery)
br
br what you most likeley want to do, is move some of those clauses like
br type_s:[*+TO+*]: and usergroup_sm:admin) out of your main q
query and
br into fq filters ... so they can be cached independently, won't
br contribute to scoring (just matching) and won't be used in
highlighting.
br
br :
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
br
br : [#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
br : org.apache.solr.servlet.SolrDispatchFilter|
br :
_ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
br : $TooManyClauses: maxClauseCount is set to 1024
br : at
org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
br ...
br : at
br :
org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
br : at
br :
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
br
br -Hoss
br
br
br
br
br
Re: maxClauseCount Exception
Thanks for the tip Hoss.
I notice that it appears sometimes and was varying because my index runs
would sometimes have different amount of docs, etc.
So all I want to do is a simple all docs with something in this field,
and to highlight the field.
Is the query expansion to all possible terms in the index really
necessary? I could have 100's of thousands of possible terms. Why should
they all become explicit query elements? Seems overkill and
underperformant.
Is there a another way with Lucene or not really?
On Thu, 2012-03-08 at 16:18 -0800, Chris Hostetter wrote:
: I am suddenly getting a maxClauseCount exception for no reason. I am
: using Solr 3.5. I have only 206 documents in my index.
Unless things have changed the reason you are seeing this is because
_highlighting_ a query (clause) like type_s:[*+TO+*] requires rewriting
it into a giant boolean query of all the terms in that field -- so even if
you only have 206 docs, if you have more then 206 values in that field in
your index, you're going to go over 1024 terms.
(you don't get this problem in a basic query, because it doens't need to
enumerate all the terms, it rewrites it to a ConstantScoreQuery)
what you most likeley want to do, is move some of those clauses like
type_s:[*+TO+*]: and usergroup_sm:admin) out of your main q query and
into fq filters ... so they can be cached independently, won't
contribute to scoring (just matching) and won't be used in highlighting.
:
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
: [#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
: org.apache.solr.servlet.SolrDispatchFilter|
: _ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
: $TooManyClauses: maxClauseCount is set to 1024
: at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
...
: at
: org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
: at
:
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
-Hoss
Re: maxClauseCount Exception
: I am suddenly getting a maxClauseCount exception for no reason. I am
: using Solr 3.5. I have only 206 documents in my index.
Unless things have changed the reason you are seeing this is because
_highlighting_ a query (clause) like type_s:[*+TO+*] requires rewriting
it into a giant boolean query of all the terms in that field -- so even if
you only have 206 docs, if you have more then 206 values in that field in
your index, you're going to go over 1024 terms.
(you don't get this problem in a basic query, because it doens't need to
enumerate all the terms, it rewrites it to a ConstantScoreQuery)
what you most likeley want to do, is move some of those clauses like
type_s:[*+TO+*]: and usergroup_sm:admin) out of your main q query and
into fq filters ... so they can be cached independently, won't
contribute to scoring (just matching) and won't be used in highlighting.
:
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
: [#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
: org.apache.solr.servlet.SolrDispatchFilter|
: _ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
: $TooManyClauses: maxClauseCount is set to 1024
: at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
...
: at
: org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
: at
:
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
-Hoss
Re: maxClauseCount Exception
Set maxBooleanClauses in your solrconfig.xml higher, default is 1024.
Your query blast this limit.
Regards
Vadim
2012/2/22 Darren Govoni dar...@ontrenet.com
Hi,
I am suddenly getting a maxClauseCount exception for no reason. I am
using Solr 3.5. I have only 206 documents in my index.
Any ideas? This is wierd.
QUERY PARAMS: [hl, hl.snippets, hl.simple.pre, hl.simple.post, fl,
hl.mergeContiguous, hl.usePhraseHighlighter, hl.requireFieldMatch,
echoParams, hl.fl, q, rows, start]|#]
[#|2012-02-22T13:40:13.129-0500|INFO|glassfish3.1.1|
org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=Thread-2;|[]
webapp=/solr3 path=/select
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
[#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
org.apache.solr.servlet.SolrDispatchFilter|
_ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:127)
at org.apache.lucene.search.ScoringRewrite
$1.addClause(ScoringRewrite.java:51)
at org.apache.lucene.search.ScoringRewrite
$1.addClause(ScoringRewrite.java:41)
at org.apache.lucene.search.ScoringRewrite
$3.collect(ScoringRewrite.java:95)
at
org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:38)
at
org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:93)
at
org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:98)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:385)
at
org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at
org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:185)
at
org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:205)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:490)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:401)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:131)
at org.apache.so
maxClauseCount Exception
Hi,
I am suddenly getting a maxClauseCount exception for no reason. I am
using Solr 3.5. I have only 206 documents in my index.
Any ideas? This is wierd.
QUERY PARAMS: [hl, hl.snippets, hl.simple.pre, hl.simple.post, fl,
hl.mergeContiguous, hl.usePhraseHighlighter, hl.requireFieldMatch,
echoParams, hl.fl, q, rows, start]|#]
[#|2012-02-22T13:40:13.129-0500|INFO|glassfish3.1.1|
org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=Thread-2;|[]
webapp=/solr3 path=/select
params={hl=truehl.snippets=4hl.simple.pre=b/bfl=*,scorehl.mergeContiguous=truehl.usePhraseHighlighter=truehl.requireFieldMatch=trueechoParams=allhl.fl=text_tq={!lucene+q.op%3DOR+df%3Dtext_t}+(+kind_s:doc+OR+kind_s:xml)+AND+(type_s:[*+TO+*])+AND+(usergroup_sm:admin)rows=20start=0wt=javabinversion=2}
hits=204 status=500 QTime=166 |#]
[#|2012-02-22T13:40:13.131-0500|SEVERE|glassfish3.1.1|
org.apache.solr.servlet.SolrDispatchFilter|
_ThreadID=22;_ThreadName=Thread-2;|org.apache.lucene.search.BooleanQuery
$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:127)
at org.apache.lucene.search.ScoringRewrite
$1.addClause(ScoringRewrite.java:51)
at org.apache.lucene.search.ScoringRewrite
$1.addClause(ScoringRewrite.java:41)
at org.apache.lucene.search.ScoringRewrite
$3.collect(ScoringRewrite.java:95)
at
org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:38)
at
org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:93)
at
org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:304)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:98)
at
org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:385)
at
org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at
org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:185)
at
org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:205)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:490)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:401)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:131)
at org.apache.so
