Re: How to debug empty ParsedQuery from Edismax Query Parser

[Kay Wrobel]

I'd like to follow up on this post here because it has become relevant to me 
now.

I have set up a debugging environment and took a deep-dive into the SOLR 7.6.0 
source code with Eclipse as my IDE of choice for this task. I have isolated the 
exact line as to where things fall apart for my two sample queries that I have 
been testing with, which are "q=a3f*" and "q=aa3f*. As you can see here, the 
only visible difference between the two search terms are that the second search 
term has two characters in succession before switching to a numerical portion.

First things first, the Extended Dismax Query Parser hands over portions of the 
parsing to the Standard Query Parser early on the the parsing process. 
Following down the rabbit hole, I ended up in 
SolrQueryParserBase.getPrefixQuery() method. On line 1173 of this method, we 
have the following statement:

termStr = analyzeIfMultitermTermText(field, termStr, 
schema.getFieldType(field));

This statement, when executing with the "a3f" search term, returns "a3f" as a 
result. However, when using "aa3f", it throws a SolrException with excatly the 
same multi-term error as shown below, only like this:
> analyzer returned too many terms for multiTerm term: aa3f

At this point, I would like to reiterate the purpose of our search: we are a 
part number house. We deal with millions of part numbers in our system and on 
our web site. A customer of ours typically searches our site with a given part 
number (or SKU if you will). Some part numbers are intelligent, and so 
customers might reduce the part number string to a portion at the beginning. 
Either way, it is *not* a typical "word" based search. Yet, the system (Drupal) 
does treat those two query fields like standard "Text" search fields. Those who 
know Drupal Commerce will recognize the Title field of a node and also possible 
the Product Variation or (SKU) field.

With that in mind, multi-term was introduced with SOLR 5, and I think this 
error (or limitation) has probably been in SOLR 5 since then. Can anyone closer 
to the matter or having struggled with this same issue chime in on the subject?

Kind regards,

Kay

> On Dec 28, 2018, at 9:57 AM, Kay Wrobel  wrote:
> 
> Here are my log entries:
> 
> SOLR 7.x (non-working)
> 2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1] 
> o.a.s.c.S.Request [collection1]  webapp=/solr path=/select 
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>  hits=0 status=0 QTime=2
> 
> SOLR 4.x (working)
> INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore; [collection1] 
> webapp=/solr path=/select 
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>  hits=32 status=0 QTime=8 
> 
> EchoParams=all did not show anything different in the resulting XML from SOLR 
> 7.x.
> 
> 
> I found out something curious yesterday. When I try to force the Standard 
> query parser on SOLR 7.x using the same query, but adding "defType=lucene" at 
> the beginning, SOLR 7 raises a SolrException with this message: "analyzer 
> returned too many terms for multiTerm term: ac6023" (full response: 
> https://pastebin.com/ijdBj4GF)
> 
> Log entry for that request:
> 2018-12-28 15:50:58.804 ERROR (qtp1769193365-15) [   x:collection1] 
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: analyzer 
> returned too many terms for multiTerm term: ac6023
>at 
> org.apache.solr.schema.TextField.analyzeMultiTerm(TextField.java:180)
>at 
> org.apache.solr.parser.SolrQueryParserBase.analyzeIfMultitermTermText(SolrQueryParserBase.java:992)
>at 
> org.apache.solr.parser.SolrQueryParserBase.getPrefixQuery(SolrQueryParserBase.java:1173)
>at 
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:781)
>at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
>at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
>at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
>at 
> org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
>at 
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:254)
>at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
>at org.apache.solr.search.QParser.getQuery(QParser.java:173)
>at 
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
>at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:279)
>at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
>at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
>at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
>at 
> 

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Kay Wrobel]

Thanks for your thoughts, Shawn. Are you a developer on SOLR?

Anyway, the configuration (solrconfig.xml) was provided by search_api_solr 
(Drupal 7 module) and is untouched. You can find it here:
https://cgit.drupalcode.org/search_api_solr/tree/solr-conf/7.x/solrconfig.xml?h=7.x-1.x

Thank you for pointing out the capital E on echoParams. I re-ran the query, but 
it doesn't change the output (at least on the surface of it).

> On Jan 2, 2019, at 1:11 PM, Shawn Heisey  wrote:
> 
> On 12/28/2018 8:57 AM, Kay Wrobel wrote:
>> Here are my log entries:
>> 
>> SOLR 7.x (non-working)
>> 2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1] 
>> o.a.s.c.S.Request [collection1] webapp=/solr path=/select 
>> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>>  hits=0 status=0 QTime=2
>> 
>> SOLR 4.x (working)
>> INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore; 
>> [collection1] webapp=/solr path=/select 
>> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>>  hits=32 status=0 QTime=8
> 
> Neither of those requests includes anything that would change from the 
> default lucene parser to edismax.  The logging *would* include all parameters 
> set by the configuration as well as those specified on the URL.
> 
> You ought to try adding "defType=edismax" to the URL parameters, or to the 
> definition of "/select" in solrconfig.xml.
> 
>> EchoParams=all did not show anything different in the resulting XML from 
>> SOLR 7.x.
> 
> The parameter requested was "echoParams" and not "EchoParams".  There *is* a 
> difference, and the latter will not work.
> 
>> I found out something curious yesterday. When I try to force the Standard 
>> query parser on SOLR 7.x using the same query, but adding "defType=lucene" 
>> at the beginning, SOLR 7 raises a SolrException with this message: "analyzer 
>> returned too many terms for multiTerm term: ac6023"
> 
> I do not know what this is about.  I did find the message in the source code. 
>  I don't understand the low-level code, and it looks to me like that section 
> of code will *always* throw an exception, which isn't very useful.
> 
> Thanks,
> Shawn


-- 

The information in this e-mail is confidential and is intended solely for 
the addressee(s). Access to this email by anyone else is unauthorized. If 
you are not an intended recipient, you may not print, save or otherwise 
store the e-mail or any of the contents thereof in electronic or physical 
form, nor copy, use or disseminate the information contained in the email.  
If you are not an intended recipient,  please notify the sender of this 
email immediately.

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Shawn Heisey]

On 12/28/2018 8:57 AM, Kay Wrobel wrote:

Here are my log entries:

SOLR 7.x (non-working)
2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1] o.a.s.c.S.Request [collection1]  
webapp=/solr path=/select 
params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
 hits=0 status=0 QTime=2

SOLR 4.x (working)
INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/select 
params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
 hits=32 status=0 QTime=8


Neither of those requests includes anything that would change from the 
default lucene parser to edismax.  The logging *would* include all 
parameters set by the configuration as well as those specified on the URL.


You ought to try adding "defType=edismax" to the URL parameters, or to 
the definition of "/select" in solrconfig.xml.



EchoParams=all did not show anything different in the resulting XML from SOLR 
7.x.


The parameter requested was "echoParams" and not "EchoParams".  There 
*is* a difference, and the latter will not work.



I found out something curious yesterday. When I try to force the Standard query parser on SOLR 7.x 
using the same query, but adding "defType=lucene" at the beginning, SOLR 7 raises a 
SolrException with this message: "analyzer returned too many terms for multiTerm term: 
ac6023"


I do not know what this is about.  I did find the message in the source 
code.  I don't understand the low-level code, and it looks to me like 
that section of code will *always* throw an exception, which isn't very 
useful.


Thanks,
Shawn

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Kay Wrobel]

Well, I was putting that info out there because I am literally hunting down 
this issue without any guidance. The real problem for still is that the Edismax 
Query Parser behaves abnormally starting with Version 5 until current giving me 
empty parsedQuery. Forcing the request through the Lucene parser was one way I 
was hoping to get to the bottom of this. Frankly, Multi-Term seems to be *the* 
new feature that was introduced in SOLR 5, and so I am jumping to conclusions 
here.

I would hate to go as low-level as debugging SOLR source just to find out what 
is going on here, but it sure seems that way. By the way, I have tried a 
multitude of other search terms (ending in *), like:
602* (works)
602K* (does NOT work)
A3F* (works!, but is also single changing characters, so...)
AC* (works, but MANY results for obvious reasons)
6023* (works)

So again, it seems that as soon as there is more than one character involved 
and a "word" is somewhat detected, the parser fails (in my specific case).

I am contemplating going down to the source-code level and debugging this 
issue; I am a programmer and should be able to understand some of it. That 
said, it seems like a very time-consuming thing to do. One last attempt right 
now is for me so change some logging level in the SOLR 7 instance and see what 
it spits out. I changed the following to "DEBUG":
org.apache.solr.search.PayloadCheckQParserPlugin
org.apache.solr.search.SurroundQParserPlugin
org.apache.solr.search.join.ChildFieldValueSourceParser

That didn't add any new information in the log file at all.


> On Jan 2, 2019, at 12:40 PM, Gus Heck  wrote:
> 
> If you mean attach a debugger, solr is just like any other java program.
> Pass in the standard java options at start up to have it listen or connect
> as usual. The port is just a TCP port so ssh tunneling the debugger port
> can bridge the gap with a remote machine (or a vpn).
> 
> That said the prior thread posts makes it sound like we are looking for a
> case where the query parser or something above it is inappropriately eating
> an exception relating to too many terms.
> 
> Did 5.x impose a new or reduced limit there?
> 
> On Wed, Jan 2, 2019, 1:20 PM Kay Wrobel  
>> Is there any way I can debug the parser? Especially, the edismax parser
>> which does *not* raise any exception but produces an empty parsedQuery?
>> Please, if anyone can help. I feel very lost and without guidance, and
>> Google search has not provided me with any help at all.
>> 
>>> On Dec 28, 2018, at 9:57 AM, Kay Wrobel  wrote:
>>> 
>>> Here are my log entries:
>>> 
>>> SOLR 7.x (non-working)
>>> 2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1]
>> o.a.s.c.S.Request [collection1]  webapp=/solr path=/select
>> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>> hits=0 status=0 QTime=2
>>> 
>>> SOLR 4.x (working)
>>> INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore;
>> [collection1] webapp=/solr path=/select
>> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>> hits=32 status=0 QTime=8
>>> 
>>> EchoParams=all did not show anything different in the resulting XML from
>> SOLR 7.x.
>>> 
>>> 
>>> I found out something curious yesterday. When I try to force the
>> Standard query parser on SOLR 7.x using the same query, but adding
>> "defType=lucene" at the beginning, SOLR 7 raises a SolrException with this
>> message: "analyzer returned too many terms for multiTerm term: ac6023"
>> (full response: https://pastebin.com/ijdBj4GF)
>>> 
>>> Log entry for that request:
>>> 2018-12-28 15:50:58.804 ERROR (qtp1769193365-15) [   x:collection1]
>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: analyzer
>> returned too many terms for multiTerm term: ac6023
>>>   at
>> org.apache.solr.schema.TextField.analyzeMultiTerm(TextField.java:180)
>>>   at
>> org.apache.solr.parser.SolrQueryParserBase.analyzeIfMultitermTermText(SolrQueryParserBase.java:992)
>>>   at
>> org.apache.solr.parser.SolrQueryParserBase.getPrefixQuery(SolrQueryParserBase.java:1173)
>>>   at
>> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:781)
>>>   at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
>>>   at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
>>>   at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
>>>   at
>> org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
>>>   at
>> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:254)
>>>   at
>> org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
>>>   at org.apache.solr.search.QParser.getQuery(QParser.java:173)
>>>   at
>> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
>>>   at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:279)
>>>   at
>> 

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Gus Heck]

If you mean attach a debugger, solr is just like any other java program.
Pass in the standard java options at start up to have it listen or connect
as usual. The port is just a TCP port so ssh tunneling the debugger port
can bridge the gap with a remote machine (or a vpn).

That said the prior thread posts makes it sound like we are looking for a
case where the query parser or something above it is inappropriately eating
an exception relating to too many terms.

Did 5.x impose a new or reduced limit there?

On Wed, Jan 2, 2019, 1:20 PM Kay Wrobel  Is there any way I can debug the parser? Especially, the edismax parser
> which does *not* raise any exception but produces an empty parsedQuery?
> Please, if anyone can help. I feel very lost and without guidance, and
> Google search has not provided me with any help at all.
>
> > On Dec 28, 2018, at 9:57 AM, Kay Wrobel  wrote:
> >
> > Here are my log entries:
> >
> > SOLR 7.x (non-working)
> > 2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1]
> o.a.s.c.S.Request [collection1]  webapp=/solr path=/select
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
> hits=0 status=0 QTime=2
> >
> > SOLR 4.x (working)
> > INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore;
> [collection1] webapp=/solr path=/select
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
> hits=32 status=0 QTime=8
> >
> > EchoParams=all did not show anything different in the resulting XML from
> SOLR 7.x.
> >
> >
> > I found out something curious yesterday. When I try to force the
> Standard query parser on SOLR 7.x using the same query, but adding
> "defType=lucene" at the beginning, SOLR 7 raises a SolrException with this
> message: "analyzer returned too many terms for multiTerm term: ac6023"
> (full response: https://pastebin.com/ijdBj4GF)
> >
> > Log entry for that request:
> > 2018-12-28 15:50:58.804 ERROR (qtp1769193365-15) [   x:collection1]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: analyzer
> returned too many terms for multiTerm term: ac6023
> >at
> org.apache.solr.schema.TextField.analyzeMultiTerm(TextField.java:180)
> >at
> org.apache.solr.parser.SolrQueryParserBase.analyzeIfMultitermTermText(SolrQueryParserBase.java:992)
> >at
> org.apache.solr.parser.SolrQueryParserBase.getPrefixQuery(SolrQueryParserBase.java:1173)
> >at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:781)
> >at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
> >at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
> >at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
> >at
> org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
> >at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:254)
> >at
> org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
> >at org.apache.solr.search.QParser.getQuery(QParser.java:173)
> >at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> >at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:279)
> >at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> >at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> >at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> >at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> >at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> >at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> >at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> >at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> >at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> >at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> >at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> >at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
> >at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> >at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> >at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> >at
> 

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Kay Wrobel]

Is there any way I can debug the parser? Especially, the edismax parser which 
does *not* raise any exception but produces an empty parsedQuery? Please, if 
anyone can help. I feel very lost and without guidance, and Google search has 
not provided me with any help at all.

> On Dec 28, 2018, at 9:57 AM, Kay Wrobel  wrote:
> 
> Here are my log entries:
> 
> SOLR 7.x (non-working)
> 2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1] 
> o.a.s.c.S.Request [collection1]  webapp=/solr path=/select 
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>  hits=0 status=0 QTime=2
> 
> SOLR 4.x (working)
> INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore; [collection1] 
> webapp=/solr path=/select 
> params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
>  hits=32 status=0 QTime=8 
> 
> EchoParams=all did not show anything different in the resulting XML from SOLR 
> 7.x.
> 
> 
> I found out something curious yesterday. When I try to force the Standard 
> query parser on SOLR 7.x using the same query, but adding "defType=lucene" at 
> the beginning, SOLR 7 raises a SolrException with this message: "analyzer 
> returned too many terms for multiTerm term: ac6023" (full response: 
> https://pastebin.com/ijdBj4GF)
> 
> Log entry for that request:
> 2018-12-28 15:50:58.804 ERROR (qtp1769193365-15) [   x:collection1] 
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: analyzer 
> returned too many terms for multiTerm term: ac6023
>at 
> org.apache.solr.schema.TextField.analyzeMultiTerm(TextField.java:180)
>at 
> org.apache.solr.parser.SolrQueryParserBase.analyzeIfMultitermTermText(SolrQueryParserBase.java:992)
>at 
> org.apache.solr.parser.SolrQueryParserBase.getPrefixQuery(SolrQueryParserBase.java:1173)
>at 
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:781)
>at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
>at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
>at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
>at 
> org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
>at 
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:254)
>at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
>at org.apache.solr.search.QParser.getQuery(QParser.java:173)
>at 
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
>at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:279)
>at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
>at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
>at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
>at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
>at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
>at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
>at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>at 
> 

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Kay Wrobel]

Here are my log entries:

SOLR 7.x (non-working)
2018-12-28 15:36:32.786 INFO  (qtp1769193365-20) [   x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/select 
params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
 hits=0 status=0 QTime=2

SOLR 4.x (working)
INFO  - 2018-12-28 15:43:41.938; org.apache.solr.core.SolrCore; [collection1] 
webapp=/solr path=/select 
params={q=ac6023*=tm_field_product^21.0=tm_title_field^8.0=all=10=xml=true}
 hits=32 status=0 QTime=8 

EchoParams=all did not show anything different in the resulting XML from SOLR 
7.x.


I found out something curious yesterday. When I try to force the Standard query 
parser on SOLR 7.x using the same query, but adding "defType=lucene" at the 
beginning, SOLR 7 raises a SolrException with this message: "analyzer returned 
too many terms for multiTerm term: ac6023" (full response: 
https://pastebin.com/ijdBj4GF)

Log entry for that request:
2018-12-28 15:50:58.804 ERROR (qtp1769193365-15) [   x:collection1] 
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: analyzer 
returned too many terms for multiTerm term: ac6023
at org.apache.solr.schema.TextField.analyzeMultiTerm(TextField.java:180)
at 
org.apache.solr.parser.SolrQueryParserBase.analyzeIfMultitermTermText(SolrQueryParserBase.java:992)
at 
org.apache.solr.parser.SolrQueryParserBase.getPrefixQuery(SolrQueryParserBase.java:1173)
at 
org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:781)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
at 
org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
at 
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:254)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
at org.apache.solr.search.QParser.getQuery(QParser.java:173)
at 
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:279)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Alexandre Rafalovitch]

EchoParams=all

May also be helpful to pinpoint differences in params from all sources,
including request handler defaults.

Regards,
Alex

On Thu, Dec 27, 2018, 8:25 PM Shawn Heisey  On 12/27/2018 10:47 AM, Kay Wrobel wrote:
> > Now starting from SOLR version 5+, I receive zero (0) results back, but
> more importantly, the Query Parser produces an empty parsedQuery.
> >
> > Here is the same query issued to SOLR 7.6.0 (current version):
> > https://pastebin.com/XcNhfdUD 
> >
> > Notice how "parsedQuery" now shows "+()"; an empty query string.
>
> I can duplicate this result on a 7.5.0 example config by sending an
> edismax query with undefined parameters for df and qf. The other
> field-related parameters for edismax are also undefined.  The following
> URL parameters with the default example config will produce that parsed
> query:
>
> q=ac6023*=edismax===on
>
> When a query is made and Solr's logging configuration is at its default
> setting, Solr will log a line into its logfile containing all of the
> parameters in the query, both those provided on the URL and those set by
> Solr's configuration (solrconfig.xml).  Can you share this log line from
> both the version that works and the version that doesn't?
>
> This is the log line created when I ran the query mentioned above:
>
> 2018-12-27 23:03:23.199 INFO  (qtp315932542-23) [   x:baz]
> o.a.s.c.S.Request [baz]  webapp=/solr path=/select
> params={q=ac6023*=edismax===on} hits=0 status=0
> QTime=0
>
> What I'm thinking is that there is a difference in the configuration of
> the two servers or the actual query being sent is different.  Either
> way, there's something different.  The two log lines that I have asked
> for are likely to be different from each other in some way that will
> explain what you're seeing.
>
> Thanks,
> Shawn
>
>

Re: How to debug empty ParsedQuery from Edismax Query Parser

[Shawn Heisey]

On 12/27/2018 10:47 AM, Kay Wrobel wrote:

Now starting from SOLR version 5+, I receive zero (0) results back, but more 
importantly, the Query Parser produces an empty parsedQuery.

Here is the same query issued to SOLR 7.6.0 (current version):
https://pastebin.com/XcNhfdUD 

Notice how "parsedQuery" now shows "+()"; an empty query string.


I can duplicate this result on a 7.5.0 example config by sending an 
edismax query with undefined parameters for df and qf. The other 
field-related parameters for edismax are also undefined.  The following 
URL parameters with the default example config will produce that parsed 
query:


q=ac6023*=edismax===on

When a query is made and Solr's logging configuration is at its default 
setting, Solr will log a line into its logfile containing all of the 
parameters in the query, both those provided on the URL and those set by 
Solr's configuration (solrconfig.xml).  Can you share this log line from 
both the version that works and the version that doesn't?


This is the log line created when I ran the query mentioned above:

2018-12-27 23:03:23.199 INFO  (qtp315932542-23) [   x:baz] 
o.a.s.c.S.Request [baz]  webapp=/solr path=/select 
params={q=ac6023*=edismax===on} hits=0 status=0 
QTime=0


What I'm thinking is that there is a difference in the configuration of 
the two servers or the actual query being sent is different.  Either 
way, there's something different.  The two log lines that I have asked 
for are likely to be different from each other in some way that will 
explain what you're seeing.


Thanks,
Shawn

Re: How to debug slow update queries?

[Shawn Heisey]

On 11/29/2017 10:29 AM, S G wrote:
> Shouldn't the slow logger print the document also which took so long?
> The /select query and /get query handlers do a good job of helping in such
> cases as the entire URL is present there (I know its GET vs POST thing).
> But we should probably print the POST request's payload for /update
> requests to debug more?

In addition to what Erick said:  The specific log message you provided
is from an update request in javabin format.  This means that the POST
content is binary -- very likely consisting of unprintable bytes, which
could cause problems if the content is logged.

Thanks,
Shawn

Re: How to debug slow update queries?

[Erick Erickson]

That would potentially mean a _lot_ of info coming out in the logs, I
know of 40M documents that have been sent for instance. I typically
batch 1,000 docs per update request, that'd mean dumping all of them.

The Solr logs should have the update request in close proximity to the
slow query message, and one of the bits of data dumped is the first 10
ids of the docs being updated along with the batch size. The last
number in the update log message is the time taken which should
correlate to 1736 in your example. That's what I usually use to try to
track updates down.

And if you're indexing 10,000 docs per batch this might be perfectly
reasonable...

Best,
Erick

On Wed, Nov 29, 2017 at 9:29 AM, S G  wrote:
> Hi,
>
> Our logs are spewing a lot of the following:
>
> org.apache.solr.core.SolrCore; slow: [my_coll_shard8_replica1]
> webapp=/solr path=/update params={wt=javabin=2} status=0 QTime=1736
>
> And the QTime is as high as 3-4 seconds in some cases.
>
> Shouldn't the slow logger print the document also which took so long?
> The /select query and /get query handlers do a good job of helping in such
> cases as the entire URL is present there (I know its GET vs POST thing).
> But we should probably print the POST request's payload for /update
> requests to debug more?
>
> Thanks
> SG

Re: Re: Antw: Re: How to Debug Solr With Eclipse

[Karthik Ramachandran]

I start Solr in my Eclipse for small test.

I have made some changes to ant build script to copy the webapp to required
location and also added eclipse launchers in this commit (
https://github.com/mrkarthik/lucene-solr/commit/d793a9b8ac0b1b4969aace4329ea5a6ddc22de16
)

Run "ant eclipse" from shell and refresh the eclipse project.  After you
could use one of the launchers to start Solr.

I have not used *-src.tgz so i am sure if this would work or not.



On Tue, Aug 15, 2017 at 3:28 PM, Erick Erickson 
wrote:

> Personally I don't start Solr inside my IDE. I use IntelliJ, but I'm
> sure Eclipse has the same capability. Have you seen:
> https://wiki.apache.org/solr/HowToConfigureEclipse?
>
> I create a "remote" configuration and fill in the blanks. Then
> starting the project like this:
>
> bin/solr start -c -z localhost:2181 -p 8983 -s ./example/cloud/node1/solr
> -a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6900"
>
> The bits after the "-a" are provided to me by IntelliJ when I create
> the remote project and specify how IntelliJ connects remotely. So I
> start Solr with the above, then attach through my IDE. At that point I
> can set breakpoints and the like, it all just works. In this
> particular case, the "suspend=y" means that Solr does _nothing_ until
> I connect.
>
> And often it's much easier to debug Solr by creating a junit test that
> does what you're interested in (or finding one of the pre-existing
> ones) and just debug that individually without even starting Solr.
>
> Best,
> Erick
>
> On Tue, Aug 15, 2017 at 10:43 AM, Rainer Gnan
>  wrote:
> > Hello Giovanni,
> >
> > I could not resolve the issue on my workingplace computer.
> > On my laptop following exactly your steps it works perfectly without
> warnings or else.
> >
> > But how to start solr correctly within eclipse (oxygen)?
> >
> > I am navigating to the subdirectory "solr/webapp/web/" an with
> right-click on index.html I select Run as Run with Jetty.
> > Run Configuration Settings are as follows:
> > - WebApp = solr/webapp/web
> > - Context Path = /solr
> > - HTTP/HTTPS = 8983
> >
> > But then I get a half of dozen of Exceptions and solr is unable to start
> ...
> > Webbrowser shows a HTTP Error 500 "java.lang.NoSuchMethodError:
> javax.servlet.ServletInputStream.isFinished()Z".
> >
> > What am I doing wrong???
> >
> > I appreciate any help or hints!
> > Rainer
> >
> >
> >
> >
> >
> > 
> >
> > Rainer Gnan
> > Bayerische Staatsbibliothek
> > Verbundzentrale des BVB
> > Referat Verbundnahe Dienste
> > 80807 München
> > Tel.: +49(0)89/28638-4445
> >
> > Fax: +49(0)89/28638-2605
> > E-Mail: rainer.g...@bsb-muenchen.de
> > 
>  Giovanni De Stefano  14.07.17 21.49 Uhr >>>
> > Hello Rainer,
> >
> > Have you found the issue?
> >
> > If not, just to be on the safe side:
> > 1) once you extracted the .tgz you get the folder `solr-6.0.0`, cd in it
> and then just
> > 2) execute `ant eclipse` and then
> > 3) in Eclipse do Import -> Existing Projects in the workspace -> select
> the `solr-6.0.0` folder (leave all options the way they are)
> > 4) wait a few minutes…it takes a while to build the whole thing, in the
> meantime it’s normal to see “errors” or “warning”…
> >
> > I hope it helps,
> > Giovanni
> >
> >
> >
> >> On 14 Jul 2017, at 16:01, Rainer Gnan 
> wrote:
> >>
> >> Hi Giovanni,
> >>
> >> thank you for this hint!
> >>
> >> The whole process (tar -xvf ..., ant compile, ant eclipse) untill
> importing the eclipse-project seems to be fine .
> >> After importing it as an existing eclipse project the project explorer
> shows an error sign on the project folder.
> >> Refreshing does not help.
> >>
> >> -> The sub-folder lucene/src is empty ...
> >>
> >> I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.
> >>
> >> Any suggestions?
> >>
> >> Cheers,
> >> Rainer
> >>
> >> 
> >> Rainer Gnan
> >> Bayerische Staatsbibliothek
> >> Verbundzentrale des BVB
> >> Referat Verbundnahe Dienste
> >> 80807 München
> >> Tel.: +49(0)89/28638-4445
> >> Fax: +49(0)89/28638-2605
> >> E-Mail: rainer.g...@bsb-muenchen.de
> >> 
> >>
> >>
> >>
> > Giovanni De Stefano  13.07.2017 19:59 >>>
> >> Hello Rainer,
> >>
> >> you have the right link: select the version you want and download the
> -src version.
> >>
> >> Once un untar the .tgz you can run `ant eclipse` from the command line
> and then import the generated project in eclipse.
> >>
> >> Please note that you will need both and and ivy installed (just start
> with ant eclipse and take it from there: the script will tell you what to
> do next).
> >>
> >> I hope it helps!
> >>
> >> Cheers,
> >> Giovanni
> >>
> >>
> >>> On 13 Jul 2017, at 19:54, govind nitk 

Re: Re: Antw: Re: How to Debug Solr With Eclipse

[Erick Erickson]

Personally I don't start Solr inside my IDE. I use IntelliJ, but I'm
sure Eclipse has the same capability. Have you seen:
https://wiki.apache.org/solr/HowToConfigureEclipse?

I create a "remote" configuration and fill in the blanks. Then
starting the project like this:

bin/solr start -c -z localhost:2181 -p 8983 -s ./example/cloud/node1/solr
-a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6900"

The bits after the "-a" are provided to me by IntelliJ when I create
the remote project and specify how IntelliJ connects remotely. So I
start Solr with the above, then attach through my IDE. At that point I
can set breakpoints and the like, it all just works. In this
particular case, the "suspend=y" means that Solr does _nothing_ until
I connect.

And often it's much easier to debug Solr by creating a junit test that
does what you're interested in (or finding one of the pre-existing
ones) and just debug that individually without even starting Solr.

Best,
Erick

On Tue, Aug 15, 2017 at 10:43 AM, Rainer Gnan
 wrote:
> Hello Giovanni,
>
> I could not resolve the issue on my workingplace computer.
> On my laptop following exactly your steps it works perfectly without warnings 
> or else.
>
> But how to start solr correctly within eclipse (oxygen)?
>
> I am navigating to the subdirectory "solr/webapp/web/" an with right-click on 
> index.html I select Run as Run with Jetty.
> Run Configuration Settings are as follows:
> - WebApp = solr/webapp/web
> - Context Path = /solr
> - HTTP/HTTPS = 8983
>
> But then I get a half of dozen of Exceptions and solr is unable to start ...
> Webbrowser shows a HTTP Error 500 "java.lang.NoSuchMethodError: 
> javax.servlet.ServletInputStream.isFinished()Z".
>
> What am I doing wrong???
>
> I appreciate any help or hints!
> Rainer
>
>
>
>
>
> 
>
> Rainer Gnan
> Bayerische Staatsbibliothek
> Verbundzentrale des BVB
> Referat Verbundnahe Dienste
> 80807 München
> Tel.: +49(0)89/28638-4445
>
> Fax: +49(0)89/28638-2605
> E-Mail: rainer.g...@bsb-muenchen.de
> 
 Giovanni De Stefano  14.07.17 21.49 Uhr >>>
> Hello Rainer,
>
> Have you found the issue?
>
> If not, just to be on the safe side:
> 1) once you extracted the .tgz you get the folder `solr-6.0.0`, cd in it and 
> then just
> 2) execute `ant eclipse` and then
> 3) in Eclipse do Import -> Existing Projects in the workspace -> select the 
> `solr-6.0.0` folder (leave all options the way they are)
> 4) wait a few minutes…it takes a while to build the whole thing, in the 
> meantime it’s normal to see “errors” or “warning”…
>
> I hope it helps,
> Giovanni
>
>
>
>> On 14 Jul 2017, at 16:01, Rainer Gnan  wrote:
>>
>> Hi Giovanni,
>>
>> thank you for this hint!
>>
>> The whole process (tar -xvf ..., ant compile, ant eclipse) untill importing 
>> the eclipse-project seems to be fine .
>> After importing it as an existing eclipse project the project explorer shows 
>> an error sign on the project folder.
>> Refreshing does not help.
>>
>> -> The sub-folder lucene/src is empty ...
>>
>> I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.
>>
>> Any suggestions?
>>
>> Cheers,
>> Rainer
>>
>> 
>> Rainer Gnan
>> Bayerische Staatsbibliothek
>> Verbundzentrale des BVB
>> Referat Verbundnahe Dienste
>> 80807 München
>> Tel.: +49(0)89/28638-4445
>> Fax: +49(0)89/28638-2605
>> E-Mail: rainer.g...@bsb-muenchen.de
>> 
>>
>>
>>
> Giovanni De Stefano  13.07.2017 19:59 >>>
>> Hello Rainer,
>>
>> you have the right link: select the version you want and download the -src 
>> version.
>>
>> Once un untar the .tgz you can run `ant eclipse` from the command line and 
>> then import the generated project in eclipse.
>>
>> Please note that you will need both and and ivy installed (just start with 
>> ant eclipse and take it from there: the script will tell you what to do 
>> next).
>>
>> I hope it helps!
>>
>> Cheers,
>> Giovanni
>>
>>
>>> On 13 Jul 2017, at 19:54, govind nitk  wrote:
>>>
>>> Hi,
>>>
>>> Solr has releases, kindly checkout to the needed one.
>>>
>>>
>>> cheers
>>>
>>> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
>>> wrote:
>>>
 Hello community,

 my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
 within Eclipse AND to test the code within a debuggable solr/lucene
 local instance - also within Eclipse.
 Searching the web led me to multiple instructions but for me no one
 works.

 The only relevant question I actually have to solve this problem is:
 Where can I download the source code for the version I want that
 includes the ANT build.xml for building an Eclipse-Project?

 The solr project page 

Antw: Re: Antw: Re: How to Debug Solr With Eclipse

[Rainer Gnan]

Hello Giovanni,

I could not resolve the issue on my workingplace computer.
On my laptop following exactly your steps it works perfectly without warnings 
or else.

But how to start solr correctly within eclipse (oxygen)?

I am navigating to the subdirectory "solr/webapp/web/" an with right-click on 
index.html I select Run as Run with Jetty.
Run Configuration Settings are as follows: 
- WebApp = solr/webapp/web
- Context Path = /solr
- HTTP/HTTPS = 8983

But then I get a half of dozen of Exceptions and solr is unable to start ...
Webbrowser shows a HTTP Error 500 "java.lang.NoSuchMethodError: 
javax.servlet.ServletInputStream.isFinished()Z".
 
What am I doing wrong???

I appreciate any help or hints!
Rainer







Rainer Gnan
Bayerische Staatsbibliothek 
Verbundzentrale des BVB
Referat Verbundnahe Dienste
80807 München
Tel.: +49(0)89/28638-4445

Fax: +49(0)89/28638-2605
E-Mail: rainer.g...@bsb-muenchen.de

>>> Giovanni De Stefano  14.07.17 21.49 Uhr >>>
Hello Rainer,

Have you found the issue?

If not, just to be on the safe side:
1) once you extracted the .tgz you get the folder `solr-6.0.0`, cd in it and 
then just 
2) execute `ant eclipse` and then 
3) in Eclipse do Import -> Existing Projects in the workspace -> select the 
`solr-6.0.0` folder (leave all options the way they are)
4) wait a few minutes…it takes a while to build the whole thing, in the 
meantime it’s normal to see “errors” or “warning”…

I hope it helps,
Giovanni



> On 14 Jul 2017, at 16:01, Rainer Gnan  wrote:
> 
> Hi Giovanni,
> 
> thank you for this hint!
> 
> The whole process (tar -xvf ..., ant compile, ant eclipse) untill importing 
> the eclipse-project seems to be fine .
> After importing it as an existing eclipse project the project explorer shows 
> an error sign on the project folder.
> Refreshing does not help.
> 
> -> The sub-folder lucene/src is empty ...
> 
> I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.
> 
> Any suggestions?
> 
> Cheers,
> Rainer
> 
> 
> Rainer Gnan
> Bayerische Staatsbibliothek 
> Verbundzentrale des BVB
> Referat Verbundnahe Dienste
> 80807 München
> Tel.: +49(0)89/28638-4445
> Fax: +49(0)89/28638-2605
> E-Mail: rainer.g...@bsb-muenchen.de
> 
> 
> 
> 
 Giovanni De Stefano  13.07.2017 19:59 >>>
> Hello Rainer,
> 
> you have the right link: select the version you want and download the -src 
> version.
> 
> Once un untar the .tgz you can run `ant eclipse` from the command line and 
> then import the generated project in eclipse.
> 
> Please note that you will need both and and ivy installed (just start with 
> ant eclipse and take it from there: the script will tell you what to do next).
> 
> I hope it helps!
> 
> Cheers,
> Giovanni
> 
> 
>> On 13 Jul 2017, at 19:54, govind nitk  wrote:
>> 
>> Hi,
>> 
>> Solr has releases, kindly checkout to the needed one.
>> 
>> 
>> cheers
>> 
>> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
>> wrote:
>> 
>>> Hello community,
>>> 
>>> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
>>> within Eclipse AND to test the code within a debuggable solr/lucene
>>> local instance - also within Eclipse.
>>> Searching the web led me to multiple instructions but for me no one
>>> works.
>>> 
>>> The only relevant question I actually have to solve this problem is:
>>> Where can I download the source code for the version I want that
>>> includes the ANT build.xml for building an Eclipse-Project?
>>> 
>>> The solr project page (http://archive.apache.org/dist/lucene/solr/)
>>> seems not to provide that.
>>> 
>>> I appreciate any hint!
>>> 
>>> Best regards
>>> Rainer
>>> 
>>> 
> 
> 

0INFO  (main) [] o.e.j.u.log Logging ini or incomplete.
680  INFO  (main) [] o.a.s.s.SolrDispatchFilter  ___  _   Welcome 
to Apache Solr™ version 6.3.0
681  INFO  (main) [] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting 
in standalone mode on port null
681  INFO  (main) [] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install 
dir: null
696  INFO  (main) [] o.a.s.s.SolrDispatchFilter |___/\___/_|_|Start 
time: 2017-08-15T17:03:32.443Z
752  INFO  (main) [] o.a.s.c.SolrResourceLoader solr home defaulted to 
'solr/' (could not find system property or JNDI)
769  INFO  (main) [] o.a.s.c.SolrXmlConfig Loading container configuration 
from C:\Users\rainer\workspace\solr-6.3.0\solr\solr.xml
775  ERROR (main) [] o.a.s.s.SolrDispatchFilter Could not start Solr. Check 
solr/home property and the logs
903  ERROR (main) [] o.a.s.c.SolrCore 
null:org.apache.solr.common.SolrException: solr.xml does not exist in 
C:\Users\rainer\workspace\solr-6.3.0\solr cannot start Solr
at 

Re: Antw: Re: How to Debug Solr With Eclipse

[Giovanni De Stefano]

Hello Rainer,

Have you found the issue?

If not, just to be on the safe side:
1) once you extracted the .tgz you get the folder `solr-6.0.0`, cd in it and 
then just 
2) execute `ant eclipse` and then 
3) in Eclipse do Import -> Existing Projects in the workspace -> select the 
`solr-6.0.0` folder (leave all options the way they are)
4) wait a few minutes…it takes a while to build the whole thing, in the 
meantime it’s normal to see “errors” or “warning”…

I hope it helps,
Giovanni



> On 14 Jul 2017, at 16:01, Rainer Gnan  wrote:
> 
> Hi Giovanni,
> 
> thank you for this hint!
> 
> The whole process (tar -xvf ..., ant compile, ant eclipse) untill importing 
> the eclipse-project seems to be fine .
> After importing it as an existing eclipse project the project explorer shows 
> an error sign on the project folder.
> Refreshing does not help.
> 
> -> The sub-folder lucene/src is empty ...
> 
> I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.
> 
> Any suggestions?
> 
> Cheers,
> Rainer
> 
> 
> Rainer Gnan
> Bayerische Staatsbibliothek 
> Verbundzentrale des BVB
> Referat Verbundnahe Dienste
> 80807 München
> Tel.: +49(0)89/28638-4445
> Fax: +49(0)89/28638-2605
> E-Mail: rainer.g...@bsb-muenchen.de
> 
> 
> 
> 
 Giovanni De Stefano  13.07.2017 19:59 >>>
> Hello Rainer,
> 
> you have the right link: select the version you want and download the -src 
> version.
> 
> Once un untar the .tgz you can run `ant eclipse` from the command line and 
> then import the generated project in eclipse.
> 
> Please note that you will need both and and ivy installed (just start with 
> ant eclipse and take it from there: the script will tell you what to do next).
> 
> I hope it helps!
> 
> Cheers,
> Giovanni
> 
> 
>> On 13 Jul 2017, at 19:54, govind nitk  wrote:
>> 
>> Hi,
>> 
>> Solr has releases, kindly checkout to the needed one.
>> 
>> 
>> cheers
>> 
>> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
>> wrote:
>> 
>>> Hello community,
>>> 
>>> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
>>> within Eclipse AND to test the code within a debuggable solr/lucene
>>> local instance - also within Eclipse.
>>> Searching the web led me to multiple instructions but for me no one
>>> works.
>>> 
>>> The only relevant question I actually have to solve this problem is:
>>> Where can I download the source code for the version I want that
>>> includes the ANT build.xml for building an Eclipse-Project?
>>> 
>>> The solr project page (http://archive.apache.org/dist/lucene/solr/)
>>> seems not to provide that.
>>> 
>>> I appreciate any hint!
>>> 
>>> Best regards
>>> Rainer
>>> 
>>> 
> 
> 

Re: Re: How to Debug Solr With Eclipse

[Erick Erickson]

Rainer:

Have you seen: https://wiki.apache.org/solr/HowToContribute? There's a
section about using Eclipse and a couple of other IDEs. I use IntelliJ
so can't help there. A number of devs use Eclipse so it should work.
Please feel free to add to the docs if you find a gotcha.

You can also pull down the source code from the Git repository and
switch to whatever branch suits your fancy. What you're doing should
work so I'm not sure what's up there.

You say: The sub-folder lucene/src is empty

I'm assuming this is in Eclipse, you do have the source when you
unpack the tgz file, right?

Best,
Erick

On Fri, Jul 14, 2017 at 7:01 AM, Rainer Gnan
 wrote:
> Hi Giovanni,
>
> thank you for this hint!
>
> The whole process (tar -xvf ..., ant compile, ant eclipse) untill importing 
> the eclipse-project seems to be fine .
> After importing it as an existing eclipse project the project explorer shows 
> an error sign on the project folder.
> Refreshing does not help.
>
> -> The sub-folder lucene/src is empty ...
>
> I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.
>
> Any suggestions?
>
> Cheers,
> Rainer
>
> 
> Rainer Gnan
> Bayerische Staatsbibliothek
> Verbundzentrale des BVB
> Referat Verbundnahe Dienste
> 80807 München
> Tel.: +49(0)89/28638-4445
> Fax: +49(0)89/28638-2605
> E-Mail: rainer.g...@bsb-muenchen.de
> 
>
>
>
 Giovanni De Stefano  13.07.2017 19:59 >>>
> Hello Rainer,
>
> you have the right link: select the version you want and download the -src 
> version.
>
> Once un untar the .tgz you can run `ant eclipse` from the command line and 
> then import the generated project in eclipse.
>
> Please note that you will need both and and ivy installed (just start with 
> ant eclipse and take it from there: the script will tell you what to do next).
>
> I hope it helps!
>
> Cheers,
> Giovanni
>
>
>> On 13 Jul 2017, at 19:54, govind nitk  wrote:
>>
>> Hi,
>>
>> Solr has releases, kindly checkout to the needed one.
>>
>>
>> cheers
>>
>> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
>> wrote:
>>
>>> Hello community,
>>>
>>> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
>>> within Eclipse AND to test the code within a debuggable solr/lucene
>>> local instance - also within Eclipse.
>>> Searching the web led me to multiple instructions but for me no one
>>> works.
>>>
>>> The only relevant question I actually have to solve this problem is:
>>> Where can I download the source code for the version I want that
>>> includes the ANT build.xml for building an Eclipse-Project?
>>>
>>> The solr project page (http://archive.apache.org/dist/lucene/solr/)
>>> seems not to provide that.
>>>
>>> I appreciate any hint!
>>>
>>> Best regards
>>> Rainer
>>>
>>>
>
>

Antw: Re: How to Debug Solr With Eclipse

[Rainer Gnan]

Hi Giovanni,

thank you for this hint!

The whole process (tar -xvf ..., ant compile, ant eclipse) untill importing the 
eclipse-project seems to be fine .
After importing it as an existing eclipse project the project explorer shows an 
error sign on the project folder.
Refreshing does not help.

-> The sub-folder lucene/src is empty ...

I am using eclipse neon, java 1.8.0_112, solr-6.6.0-src.tgz.

Any suggestions?

Cheers,
Rainer


Rainer Gnan
Bayerische Staatsbibliothek 
Verbundzentrale des BVB
Referat Verbundnahe Dienste
80807 München
Tel.: +49(0)89/28638-4445
Fax: +49(0)89/28638-2605
E-Mail: rainer.g...@bsb-muenchen.de




>>> Giovanni De Stefano  13.07.2017 19:59 >>>
Hello Rainer,

you have the right link: select the version you want and download the -src 
version.

Once un untar the .tgz you can run `ant eclipse` from the command line and then 
import the generated project in eclipse.

Please note that you will need both and and ivy installed (just start with ant 
eclipse and take it from there: the script will tell you what to do next).

I hope it helps!

Cheers,
Giovanni


> On 13 Jul 2017, at 19:54, govind nitk  wrote:
> 
> Hi,
> 
> Solr has releases, kindly checkout to the needed one.
> 
> 
> cheers
> 
> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
> wrote:
> 
>> Hello community,
>> 
>> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
>> within Eclipse AND to test the code within a debuggable solr/lucene
>> local instance - also within Eclipse.
>> Searching the web led me to multiple instructions but for me no one
>> works.
>> 
>> The only relevant question I actually have to solve this problem is:
>> Where can I download the source code for the version I want that
>> includes the ANT build.xml for building an Eclipse-Project?
>> 
>> The solr project page (http://archive.apache.org/dist/lucene/solr/)
>> seems not to provide that.
>> 
>> I appreciate any hint!
>> 
>> Best regards
>> Rainer
>> 
>> 

Re: How to Debug Solr With Eclipse

[Lars Karlsson]

Another option that I normally find easier is attaching to existing Solr
instance and do remote debugging.
It require changing the JVM options and restarting Solr, then from eclipse
choose Remote Java Application.

On Thu, 13 Jul 2017 at 19:50, Rainer Gnan 
wrote:

> Hello community,
>
> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
> within Eclipse AND to test the code within a debuggable solr/lucene
> local instance - also within Eclipse.
> Searching the web led me to multiple instructions but for me no one
> works.
>
> The only relevant question I actually have to solve this problem is:
> Where can I download the source code for the version I want that
> includes the ANT build.xml for building an Eclipse-Project?
>
> The solr project page (http://archive.apache.org/dist/lucene/solr/)
> seems not to provide that.
>
> I appreciate any hint!
>
> Best regards
> Rainer
>
>

Re: How to Debug Solr With Eclipse

[Giovanni De Stefano]

Hello Rainer,

you have the right link: select the version you want and download the -src 
version.

Once un untar the .tgz you can run `ant eclipse` from the command line and then 
import the generated project in eclipse.

Please note that you will need both and and ivy installed (just start with ant 
eclipse and take it from there: the script will tell you what to do next).

I hope it helps!

Cheers,
Giovanni


> On 13 Jul 2017, at 19:54, govind nitk  wrote:
> 
> Hi,
> 
> Solr has releases, kindly checkout to the needed one.
> 
> 
> cheers
> 
> On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
> wrote:
> 
>> Hello community,
>> 
>> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
>> within Eclipse AND to test the code within a debuggable solr/lucene
>> local instance - also within Eclipse.
>> Searching the web led me to multiple instructions but for me no one
>> works.
>> 
>> The only relevant question I actually have to solve this problem is:
>> Where can I download the source code for the version I want that
>> includes the ANT build.xml for building an Eclipse-Project?
>> 
>> The solr project page (http://archive.apache.org/dist/lucene/solr/)
>> seems not to provide that.
>> 
>> I appreciate any hint!
>> 
>> Best regards
>> Rainer
>> 
>> 

Re: How to Debug Solr With Eclipse

[govind nitk]

Hi,

Solr has releases, kindly checkout to the needed one.


cheers

On Thu, Jul 13, 2017 at 11:20 PM, Rainer Gnan 
wrote:

> Hello community,
>
> my aim is to develop solr custom code (e.g. UpdateRequestProcessor)
> within Eclipse AND to test the code within a debuggable solr/lucene
> local instance - also within Eclipse.
> Searching the web led me to multiple instructions but for me no one
> works.
>
> The only relevant question I actually have to solve this problem is:
> Where can I download the source code for the version I want that
> includes the ANT build.xml for building an Eclipse-Project?
>
> The solr project page (http://archive.apache.org/dist/lucene/solr/)
> seems not to provide that.
>
> I appreciate any hint!
>
> Best regards
> Rainer
>
>

Re: how to debug solr performance degradation

[Shawn Heisey]

On 2/27/2015 12:51 PM, Tang, Rebecca wrote:
 Thank you guys for all the suggestions and help! I'Ve identified the main
 culprit with debug=timing.  It was the mlt component.  After I removed it,
 the speed of the query went back to reasonable.  Another culprit is the
 expand component, but I can't remove it.  We've downgraded our amazon
 instance to 60G mem with general purpose SSD and the performance is pretty
 good.  It's only 70 cents/hr versus 2.80/hr for the 244G mem instance :)

 I also added all the suggested JMV parameters.  Now I have a gc.log that I
 dig into.

 One thing I would like to understand is how memory is managed by solr.

 If I do 'top -u solr', I see something like this:

 Mem:  62920240k total, 62582524k used,   337716k free,   133360k buffers
 Swap:0k total,0k used,0k free, 54500892k cached

   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
 
  4266 solr  20   0  192g 5.1g 854m S  0.0  8.4  37:09.97 java

 There are two things:
 1) Mem: 62920240k total, 62582524k used. I think this is what the solr
 admin physical memory bar graph reports on.  Can I assume that most of
 the mem is used for loading part of the index?

 2) And then there's the VIRT 192g and RES 5.1g.  What is the 5.1 RES
 (physical memory) that is used by solr?

The total and used values from top refer to *all* memory in the
entire machine, and it does match the physical memory graph in the
admin UI.  If you notice that the cached value is 54GB, that's where
most of the memory usage is actually happening.  This is the OS disk
cache -- the OS is automatically using extra memory to cache data on the
disk.  You are only caching about a third of your index, which may not
be enough for good performance, especially with complex queries.

The VIRT (virtual) and RES (resident) values are describing how Java is
using memory from the OS point of view.  The java process has allocated
5.1GB of RAM for the heap and all other memory structures.  The VIRT
number is the total amount of *address space* (virtual memory, not
actual memory) that the process has allocated.  For Solr, this will
typically be (approximately) the size of all your indexes plus the RES
and SHR values.

Solr (Lucene) uses the mmap functionality in the operating system for
all disk access by default (configurable) -- this means that it maps the
file on the disk into virtual memory.  This makes it so that a program
doesn't need to use disk I/O calls to access the data ... it just
pretends that the file is sitting in memory.  The operating system takes
care of translating those memory reads and writes into disk access.  All
memory that is not explicitly allocated to a program is automatically
used to cache that disk access -- this is the cached number from top
that I already mentioned.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
http://en.wikipedia.org/wiki/Page_cache

Thanks,
Shawn

Re: how to debug solr performance degradation

[Tang, Rebecca]

Thank you guys for all the suggestions and help! I'Ve identified the main
culprit with debug=timing.  It was the mlt component.  After I removed it,
the speed of the query went back to reasonable.  Another culprit is the
expand component, but I can't remove it.  We've downgraded our amazon
instance to 60G mem with general purpose SSD and the performance is pretty
good.  It's only 70 cents/hr versus 2.80/hr for the 244G mem instance :)

I also added all the suggested JMV parameters.  Now I have a gc.log that I
dig into.

One thing I would like to understand is how memory is managed by solr.

If I do 'top -u solr', I see something like this:

Mem:  62920240k total, 62582524k used,   337716k free,   133360k buffers
Swap:0k total,0k used,0k free, 54500892k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND

 4266 solr  20   0  192g 5.1g 854m S  0.0  8.4  37:09.97 java

There are two things:
1) Mem: 62920240k total, 62582524k used. I think this is what the solr
admin physical memory bar graph reports on.  Can I assume that most of
the mem is used for loading part of the index?

2) And then there's the VIRT 192g and RES 5.1g.  What is the 5.1 RES
(physical memory) that is used by solr?




Rebecca Tang
Applications Developer, UCSF CKM
Industry Documents Digital Libraries
E: rebecca.t...@ucsf.edu





On 2/25/15 7:57 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote:

Lots of suggestions here already.  +1 for those JVM params from Boogie and
for looking at JMX.
Rebecca, try SPM http://sematext.com/spm (will look at JMX for you,
among
other things), it may save you time figuring out
JVM/heap/memory/performance issues.  If you can't tell what's slow via
SPM,
we can have a look at your metrics (charts are sharable) and may be able
to
help you faster than guessing.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr  Elasticsearch Support * http://sematext.com/


On Wed, Feb 25, 2015 at 4:27 PM, Erick Erickson erickerick...@gmail.com
wrote:

 Before diving in too deeply, try attaching debug=timing to the query.
 Near the bottom of the response there'll be a list of the time taken
 by each _component_. So there'll be separate entries for query,
 highlighting, etc.

 This may not show any surprises, you might be spending all your time
 scoring. But it's worth doing as a check and might save you from going
 down some dead-ends. I mean if your query winds up spending 80% of its
 time in the highlighter you know where to start looking..

 Best,
 Erick


 On Wed, Feb 25, 2015 at 12:01 PM, Boogie Shafer
 boogie.sha...@proquest.com wrote:
  rebecca,
 
  you probably need to dig into your queries, but if you want to
 force/preload the index into memory you could try doing something like
 
  cat `find /path/to/solr/index`  /dev/null
 
 
  if you haven't already reviewed the following, you might take a look
here
  https://wiki.apache.org/solr/SolrPerformanceProblems
 
  perhaps going back to a very vanilla/default solr configuration and
 building back up from that baseline to better isolate what might
specific
 setting be impacting your environment
 
  
  From: Tang, Rebecca rebecca.t...@ucsf.edu
  Sent: Wednesday, February 25, 2015 11:44
  To: solr-user@lucene.apache.org
  Subject: RE: how to debug solr performance degradation
 
  Sorry, I should have been more specific.
 
  I was referring to the solr admin UI page. Today we started up an AWS
  instance with 240 G of memory to see if we fit all of our index
(183G) in
  the memory and have enough for the JMV, could it improve the
performance.
 
  I attached the admin UI screen shot with the email.
 
  The top bar is ³Physical Memory² and we have 240.24 GB, but only 4%
9.52
  GB is used.
 
  The next bar is Swap Space and it¹s at 0.00 MB.
 
  The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.
 
  My understanding is that when Solr starts up, it reserves some memory
for
  the JVM, and then it tries to use up as much of the remaining physical
  memory as possible.  And I used to see the physical memory at anywhere
  between 70% to 90+%.  Is this understanding correct?
 
  And now, even with 240G of memory, our index is performing at 10 - 20
  seconds for a query.  Granted that our queries have fq¹s and
highlighting
  and faceting, I think with a machine this powerful I should be able to
 get
  the queries executed under 5 seconds.
 
  This is what we send to Solr:
  q=(phillip%20morris)
  wt=json
  start=0
  rows=50
  facet=true
  facet.mincount=0
  facet.pivot=industry,collection_facet
  facet.pivot=availability_facet,availabilitystatus_facet
  facet.field=dddate
 
 
fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blan
k%
 
 
20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20
be
 
 
gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20fold
er
 
 
%20end%22%20OR%20dt%3A

Re: how to debug solr performance degradation

[Boogie Shafer]

rebecca,

you probably need to dig into your queries, but if you want to force/preload 
the index into memory you could try doing something like

cat `find /path/to/solr/index`  /dev/null


if you haven't already reviewed the following, you might take a look here
https://wiki.apache.org/solr/SolrPerformanceProblems

perhaps going back to a very vanilla/default solr configuration and building 
back up from that baseline to better isolate what might specific setting be 
impacting your environment


From: Tang, Rebecca rebecca.t...@ucsf.edu
Sent: Wednesday, February 25, 2015 11:44
To: solr-user@lucene.apache.org
Subject: RE: how to debug solr performance degradation

Sorry, I should have been more specific.

I was referring to the solr admin UI page. Today we started up an AWS
instance with 240 G of memory to see if we fit all of our index (183G) in
the memory and have enough for the JMV, could it improve the performance.

I attached the admin UI screen shot with the email.

The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
GB is used.

The next bar is Swap Space and it¹s at 0.00 MB.

The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.

My understanding is that when Solr starts up, it reserves some memory for
the JVM, and then it tries to use up as much of the remaining physical
memory as possible.  And I used to see the physical memory at anywhere
between 70% to 90+%.  Is this understanding correct?

And now, even with 240G of memory, our index is performing at 10 - 20
seconds for a query.  Granted that our queries have fq¹s and highlighting
and faceting, I think with a machine this powerful I should be able to get
the queries executed under 5 seconds.

This is what we send to Solr:
q=(phillip%20morris)
wt=json
start=0
rows=50
facet=true
facet.mincount=0
facet.pivot=industry,collection_facet
facet.pivot=availability_facet,availabilitystatus_facet
facet.field=dddate
fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
%20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
22%20OR%20dt%3A%22tab%20sheet%22))
facet.field=dt_facet
facet.field=brd_facet
facet.field=dg_facet
hl=true
hl.simple.pre=%3Ch1%3E
hl.simple.post=%3C%2Fh1%3E
hl.requireFieldMatch=false
hl.preserveMulti=true
hl.fl=ot,ti
f.ot.hl.fragsize=300
f.ot.hl.alternateField=ot
f.ot.hl.maxAlternateFieldLength=300
f.ti.hl.fragsize=300
f.ti.hl.alternateField=ti
f.ti.hl.maxAlternateFieldLength=300
fq={!collapse%20field=signature}
expand=true
sort=score+desc,availability_facet+asc


My guess is that it¹s performing so badly because it¹s only using 4% of
the memory? And searches require disk access.


Rebecca

From: Shawn Heisey [apa...@elyograg.org]
Sent: Tuesday, February 24, 2015 5:23 PM
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

I would like to know what memory numbers in which program you are
looking at, and why you believe those numbers are a problem.

The JVM has a very different view of memory than the operating system.
Numbers in top mean different things than numbers on the dashboard of
the admin UI, or the numbers in jconsole.  If you're on Windows, then
replace top with task manager, process explorer, resource monitor, etc.

Please provide as many details as you can about the things you are
looking at.

Thanks,
Shawn

RE: how to debug solr performance degradation

[Toke Eskildsen]

Unfortunately (or luckily, depending on view), attachments does not work with 
this mailing list. You'll have to upload it somewhere and provide an URL. It is 
quite hard _not_ to get your whole index into disk cache, so my guess is that 
it will get there eventually. Just to check: If you re-issue your queries, does 
the response time change? If not, then disk caching is not the problem.

Anyway, with your new information, I would say that pivot faceting is the 
culprit. Does the timing tests in 
https://issues.apache.org/jira/browse/SOLR-6803 line up with the cardinalities 
of your fields?

My next step would be to disable parts of the query (highlight, faceting and 
collapsing one at a time) to check which part is the heaviest.

- Toke Eskildsen

From: Tang, Rebecca [rebecca.t...@ucsf.edu]
Sent: 25 February 2015 20:44
To: solr-user@lucene.apache.org
Subject: RE: how to debug solr performance degradation

Sorry, I should have been more specific.

I was referring to the solr admin UI page. Today we started up an AWS
instance with 240 G of memory to see if we fit all of our index (183G) in
the memory and have enough for the JMV, could it improve the performance.

I attached the admin UI screen shot with the email.

The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
GB is used.

The next bar is Swap Space and it¹s at 0.00 MB.

The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.

My understanding is that when Solr starts up, it reserves some memory for
the JVM, and then it tries to use up as much of the remaining physical
memory as possible.  And I used to see the physical memory at anywhere
between 70% to 90+%.  Is this understanding correct?

And now, even with 240G of memory, our index is performing at 10 - 20
seconds for a query.  Granted that our queries have fq¹s and highlighting
and faceting, I think with a machine this powerful I should be able to get
the queries executed under 5 seconds.

This is what we send to Solr:
q=(phillip%20morris)
wt=json
start=0
rows=50
facet=true
facet.mincount=0
facet.pivot=industry,collection_facet
facet.pivot=availability_facet,availabilitystatus_facet
facet.field=dddate
fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
%20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
22%20OR%20dt%3A%22tab%20sheet%22))
facet.field=dt_facet
facet.field=brd_facet
facet.field=dg_facet
hl=true
hl.simple.pre=%3Ch1%3E
hl.simple.post=%3C%2Fh1%3E
hl.requireFieldMatch=false
hl.preserveMulti=true
hl.fl=ot,ti
f.ot.hl.fragsize=300
f.ot.hl.alternateField=ot
f.ot.hl.maxAlternateFieldLength=300
f.ti.hl.fragsize=300
f.ti.hl.alternateField=ti
f.ti.hl.maxAlternateFieldLength=300
fq={!collapse%20field=signature}
expand=true
sort=score+desc,availability_facet+asc


My guess is that it¹s performing so badly because it¹s only using 4% of
the memory? And searches require disk access.


Rebecca

From: Shawn Heisey [apa...@elyograg.org]
Sent: Tuesday, February 24, 2015 5:23 PM
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

I would like to know what memory numbers in which program you are
looking at, and why you believe those numbers are a problem.

The JVM has a very different view of memory than the operating system.
Numbers in top mean different things than numbers on the dashboard of
the admin UI, or the numbers in jconsole.  If you're on Windows, then
replace top with task manager, process explorer, resource monitor, etc.

Please provide as many details as you can about the things you are
looking at.

Thanks,
Shawn

RE: how to debug solr performance degradation

[Tang, Rebecca]

Sorry, I should have been more specific.

I was referring to the solr admin UI page. Today we started up an AWS
instance with 240 G of memory to see if we fit all of our index (183G) in
the memory and have enough for the JMV, could it improve the performance.

I attached the admin UI screen shot with the email.

The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
GB is used.

The next bar is Swap Space and it¹s at 0.00 MB.

The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.

My understanding is that when Solr starts up, it reserves some memory for
the JVM, and then it tries to use up as much of the remaining physical
memory as possible.  And I used to see the physical memory at anywhere
between 70% to 90+%.  Is this understanding correct?

And now, even with 240G of memory, our index is performing at 10 - 20
seconds for a query.  Granted that our queries have fq¹s and highlighting
and faceting, I think with a machine this powerful I should be able to get
the queries executed under 5 seconds.

This is what we send to Solr:
q=(phillip%20morris)
wt=json
start=0
rows=50
facet=true
facet.mincount=0
facet.pivot=industry,collection_facet
facet.pivot=availability_facet,availabilitystatus_facet
facet.field=dddate
fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
%20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
22%20OR%20dt%3A%22tab%20sheet%22))
facet.field=dt_facet
facet.field=brd_facet
facet.field=dg_facet
hl=true
hl.simple.pre=%3Ch1%3E
hl.simple.post=%3C%2Fh1%3E
hl.requireFieldMatch=false
hl.preserveMulti=true
hl.fl=ot,ti
f.ot.hl.fragsize=300
f.ot.hl.alternateField=ot
f.ot.hl.maxAlternateFieldLength=300
f.ti.hl.fragsize=300
f.ti.hl.alternateField=ti
f.ti.hl.maxAlternateFieldLength=300
fq={!collapse%20field=signature}
expand=true
sort=score+desc,availability_facet+asc


My guess is that it¹s performing so badly because it¹s only using 4% of
the memory? And searches require disk access.


Rebecca

From: Shawn Heisey [apa...@elyograg.org]
Sent: Tuesday, February 24, 2015 5:23 PM
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

I would like to know what memory numbers in which program you are
looking at, and why you believe those numbers are a problem.

The JVM has a very different view of memory than the operating system.
Numbers in top mean different things than numbers on the dashboard of
the admin UI, or the numbers in jconsole.  If you're on Windows, then
replace top with task manager, process explorer, resource monitor, etc.

Please provide as many details as you can about the things you are
looking at.

Thanks,
Shawn

Re: how to debug solr performance degradation

[Erick Erickson]

Before diving in too deeply, try attaching debug=timing to the query.
Near the bottom of the response there'll be a list of the time taken
by each _component_. So there'll be separate entries for query,
highlighting, etc.

This may not show any surprises, you might be spending all your time
scoring. But it's worth doing as a check and might save you from going
down some dead-ends. I mean if your query winds up spending 80% of its
time in the highlighter you know where to start looking..

Best,
Erick


On Wed, Feb 25, 2015 at 12:01 PM, Boogie Shafer
boogie.sha...@proquest.com wrote:
 rebecca,

 you probably need to dig into your queries, but if you want to force/preload 
 the index into memory you could try doing something like

 cat `find /path/to/solr/index`  /dev/null


 if you haven't already reviewed the following, you might take a look here
 https://wiki.apache.org/solr/SolrPerformanceProblems

 perhaps going back to a very vanilla/default solr configuration and building 
 back up from that baseline to better isolate what might specific setting be 
 impacting your environment

 
 From: Tang, Rebecca rebecca.t...@ucsf.edu
 Sent: Wednesday, February 25, 2015 11:44
 To: solr-user@lucene.apache.org
 Subject: RE: how to debug solr performance degradation

 Sorry, I should have been more specific.

 I was referring to the solr admin UI page. Today we started up an AWS
 instance with 240 G of memory to see if we fit all of our index (183G) in
 the memory and have enough for the JMV, could it improve the performance.

 I attached the admin UI screen shot with the email.

 The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
 GB is used.

 The next bar is Swap Space and it¹s at 0.00 MB.

 The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.

 My understanding is that when Solr starts up, it reserves some memory for
 the JVM, and then it tries to use up as much of the remaining physical
 memory as possible.  And I used to see the physical memory at anywhere
 between 70% to 90+%.  Is this understanding correct?

 And now, even with 240G of memory, our index is performing at 10 - 20
 seconds for a query.  Granted that our queries have fq¹s and highlighting
 and faceting, I think with a machine this powerful I should be able to get
 the queries executed under 5 seconds.

 This is what we send to Solr:
 q=(phillip%20morris)
 wt=json
 start=0
 rows=50
 facet=true
 facet.mincount=0
 facet.pivot=industry,collection_facet
 facet.pivot=availability_facet,availabilitystatus_facet
 facet.field=dddate
 fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
 20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
 gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
 %20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
 et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
 22%20OR%20dt%3A%22tab%20sheet%22))
 facet.field=dt_facet
 facet.field=brd_facet
 facet.field=dg_facet
 hl=true
 hl.simple.pre=%3Ch1%3E
 hl.simple.post=%3C%2Fh1%3E
 hl.requireFieldMatch=false
 hl.preserveMulti=true
 hl.fl=ot,ti
 f.ot.hl.fragsize=300
 f.ot.hl.alternateField=ot
 f.ot.hl.maxAlternateFieldLength=300
 f.ti.hl.fragsize=300
 f.ti.hl.alternateField=ti
 f.ti.hl.maxAlternateFieldLength=300
 fq={!collapse%20field=signature}
 expand=true
 sort=score+desc,availability_facet+asc


 My guess is that it¹s performing so badly because it¹s only using 4% of
 the memory? And searches require disk access.


 Rebecca
 
 From: Shawn Heisey [apa...@elyograg.org]
 Sent: Tuesday, February 24, 2015 5:23 PM
 To: solr-user@lucene.apache.org
 Subject: Re: how to debug solr performance degradation

 On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

 I would like to know what memory numbers in which program you are
 looking at, and why you believe those numbers are a problem.

 The JVM has a very different view of memory than the operating system.
 Numbers in top mean different things than numbers on the dashboard of
 the admin UI, or the numbers in jconsole.  If you're on Windows, then
 replace top with task manager, process explorer, resource monitor, etc.

 Please provide as many details as you can about the things you are
 looking at.

 Thanks,
 Shawn

Re: how to debug solr performance degradation

[Otis Gospodnetic]

Lots of suggestions here already.  +1 for those JVM params from Boogie and
for looking at JMX.
Rebecca, try SPM http://sematext.com/spm (will look at JMX for you, among
other things), it may save you time figuring out
JVM/heap/memory/performance issues.  If you can't tell what's slow via SPM,
we can have a look at your metrics (charts are sharable) and may be able to
help you faster than guessing.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr  Elasticsearch Support * http://sematext.com/


On Wed, Feb 25, 2015 at 4:27 PM, Erick Erickson erickerick...@gmail.com
wrote:

 Before diving in too deeply, try attaching debug=timing to the query.
 Near the bottom of the response there'll be a list of the time taken
 by each _component_. So there'll be separate entries for query,
 highlighting, etc.

 This may not show any surprises, you might be spending all your time
 scoring. But it's worth doing as a check and might save you from going
 down some dead-ends. I mean if your query winds up spending 80% of its
 time in the highlighter you know where to start looking..

 Best,
 Erick


 On Wed, Feb 25, 2015 at 12:01 PM, Boogie Shafer
 boogie.sha...@proquest.com wrote:
  rebecca,
 
  you probably need to dig into your queries, but if you want to
 force/preload the index into memory you could try doing something like
 
  cat `find /path/to/solr/index`  /dev/null
 
 
  if you haven't already reviewed the following, you might take a look here
  https://wiki.apache.org/solr/SolrPerformanceProblems
 
  perhaps going back to a very vanilla/default solr configuration and
 building back up from that baseline to better isolate what might specific
 setting be impacting your environment
 
  
  From: Tang, Rebecca rebecca.t...@ucsf.edu
  Sent: Wednesday, February 25, 2015 11:44
  To: solr-user@lucene.apache.org
  Subject: RE: how to debug solr performance degradation
 
  Sorry, I should have been more specific.
 
  I was referring to the solr admin UI page. Today we started up an AWS
  instance with 240 G of memory to see if we fit all of our index (183G) in
  the memory and have enough for the JMV, could it improve the performance.
 
  I attached the admin UI screen shot with the email.
 
  The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
  GB is used.
 
  The next bar is Swap Space and it¹s at 0.00 MB.
 
  The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.
 
  My understanding is that when Solr starts up, it reserves some memory for
  the JVM, and then it tries to use up as much of the remaining physical
  memory as possible.  And I used to see the physical memory at anywhere
  between 70% to 90+%.  Is this understanding correct?
 
  And now, even with 240G of memory, our index is performing at 10 - 20
  seconds for a query.  Granted that our queries have fq¹s and highlighting
  and faceting, I think with a machine this powerful I should be able to
 get
  the queries executed under 5 seconds.
 
  This is what we send to Solr:
  q=(phillip%20morris)
  wt=json
  start=0
  rows=50
  facet=true
  facet.mincount=0
  facet.pivot=industry,collection_facet
  facet.pivot=availability_facet,availabilitystatus_facet
  facet.field=dddate
 
 fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
 
 20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
 
 gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
 
 %20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
 
 et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
  22%20OR%20dt%3A%22tab%20sheet%22))
  facet.field=dt_facet
  facet.field=brd_facet
  facet.field=dg_facet
  hl=true
  hl.simple.pre=%3Ch1%3E
  hl.simple.post=%3C%2Fh1%3E
  hl.requireFieldMatch=false
  hl.preserveMulti=true
  hl.fl=ot,ti
  f.ot.hl.fragsize=300
  f.ot.hl.alternateField=ot
  f.ot.hl.maxAlternateFieldLength=300
  f.ti.hl.fragsize=300
  f.ti.hl.alternateField=ti
  f.ti.hl.maxAlternateFieldLength=300
  fq={!collapse%20field=signature}
  expand=true
  sort=score+desc,availability_facet+asc
 
 
  My guess is that it¹s performing so badly because it¹s only using 4% of
  the memory? And searches require disk access.
 
 
  Rebecca
  
  From: Shawn Heisey [apa...@elyograg.org]
  Sent: Tuesday, February 24, 2015 5:23 PM
  To: solr-user@lucene.apache.org
  Subject: Re: how to debug solr performance degradation
 
  On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
  We gave the machine 180G mem to see if it improves performance.
 However,
  after we increased the memory, Solr started using only 5% of the
 physical
  memory.  It has always used 90-something%.
 
  What could be causing solr to not grab all the physical memory (grabbing
  so little of the physical memory)?
 
  I would like to know what memory numbers in which program you are
  looking at, and why you

RE: how to debug solr performance degradation

[Toke Eskildsen]

Tang, Rebecca [rebecca.t...@ucsf.edu] wrote:
[12-15 second response time instead of 0-3]
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

The usual culprit is memory (if you are using spinning drive as your storage). 
It appears that you have enough raw memory though. Could you check how much 
memory the machine has free for disk caching? If it is a relative small amount, 
let's say below 50GB, then please provide a breakdown of what the memory is 
used for (very large JVM heap for example).

 I want to pinpoint where the performance issue is coming from.  Could I have 
 some suggestions/help on how to benchmark/debug solr performance issues.

Rough checking of IOWait and CPU load is a fine starting point. If if is CPU 
load then you can turn on debug in Solr admin, which should tell you where the 
time is spend resolving the queries. It it is IOWait then ensure a lot of free 
memory for disk cache and/or improve your storage speed (SSDs instead of 
spinning drives, local storage instead of remote).

- Toke Eskildsen, State and University Library, Denmark.

Re: how to debug solr performance degradation

[Shawn Heisey]

On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere 
 between 0-3 seconds to complete any query), but today I noticed that the 
 performance is very bad (queries take between 12 – 15 seconds).

 I haven't updated the solr index configuration (schema.xml/solrconfig.xml) 
 lately.  All that's changed is the data — every month, I rebuild the solr 
 index from scratch and deploy it to the box.  We will eventually go to 
 incremental builds. But for now, all indexes are built from scratch.

 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

The bottom line on this problem, and I'm sure it's not something you're
going to want to hear:  You don't have enough memory available to cache
your index.  I'd plan on at least 192GB of RAM for an index this size,
and 256GB would be better.

Depending on the exact index schema, the nature of your queries, and how
large your Java heap for Solr is, 100GB of RAM could be enough for good
performance on an index that size ... or it might be nowhere near
enough.  I would imagine that one of two things is true here, possibly
both:  1) Your queries are very complex and involve accessing a very
large percentage of the index data.  2) Your Java heap is enormous,
leaving very little RAM for the OS to automatically cache the index.

Adding more memory to the machine, if that's possible, might fix some of
the problems.  You can find a discussion of the problem here:

http://wiki.apache.org/solr/SolrPerformanceProblems

If you have any questions after reading that wiki article, feel free to
ask them.

Thanks,
Shawn

Re: how to debug solr performance degradation

[Shawn Heisey]

On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

I would like to know what memory numbers in which program you are
looking at, and why you believe those numbers are a problem.

The JVM has a very different view of memory than the operating system. 
Numbers in top mean different things than numbers on the dashboard of
the admin UI, or the numbers in jconsole.  If you're on Windows, then
replace top with task manager, process explorer, resource monitor, etc.

Please provide as many details as you can about the things you are
looking at.

Thanks,
Shawn

Re: how to debug solr performance degradation

[Erick Erickson]

Be careful what you think is being used by Solr since Lucene uses
MMapDirectories under the covers, and this means you might be seeing
virtual memory. See Uwe's excellent blog here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Best,
Erick

On Tue, Feb 24, 2015 at 5:02 PM, Walter Underwood wun...@wunderwood.org wrote:
 The other memory is used by the OS as file buffers. All the important parts 
 of the on-disk search index are buffered in memory. When the Solr process 
 wants a block, it is already right there, no delays for disk access.

 wunder
 Walter Underwood
 wun...@wunderwood.org
 http://observer.wunderwood.org/  (my blog)


 On Feb 24, 2015, at 4:45 PM, Tang, Rebecca rebecca.t...@ucsf.edu wrote:

 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?

 Rebecca Tang
 Applications Developer, UCSF CKM
 Industry Documents Digital Libraries
 E: rebecca.t...@ucsf.edu

 On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
 between 0-3 seconds to complete any query), but today I noticed that the
 performance is very bad (queries take between 12 ­ 15 seconds).

 I haven't updated the solr index configuration
 (schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
 every month, I rebuild the solr index from scratch and deploy it to the
 box.  We will eventually go to incremental builds. But for now, all
 indexes are built from scratch.

 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

 The bottom line on this problem, and I'm sure it's not something you're
 going to want to hear:  You don't have enough memory available to cache
 your index.  I'd plan on at least 192GB of RAM for an index this size,
 and 256GB would be better.

 Depending on the exact index schema, the nature of your queries, and how
 large your Java heap for Solr is, 100GB of RAM could be enough for good
 performance on an index that size ... or it might be nowhere near
 enough.  I would imagine that one of two things is true here, possibly
 both:  1) Your queries are very complex and involve accessing a very
 large percentage of the index data.  2) Your Java heap is enormous,
 leaving very little RAM for the OS to automatically cache the index.

 Adding more memory to the machine, if that's possible, might fix some of
 the problems.  You can find a discussion of the problem here:

 http://wiki.apache.org/solr/SolrPerformanceProblems

 If you have any questions after reading that wiki article, feel free to
 ask them.

 Thanks,
 Shawn

Re: how to debug solr performance degradation

[François Schiettecatte]

Rebecca

You don’t want to give all the memory to the JVM. You want to give it just 
enough for it to work optimally and leave the rest of the memory for the OS to 
use for caching data. Giving the JVM too much memory can result in worse 
performance because of GC. There is no magic formula to figuring out the memory 
allocation for the JVM, that is very dependent on the workload. In your case I 
would start with 5GB, and increment by 5GB with each run.

I also use these settings for the JVM

-XX:+UseG1GC -Xms1G -Xmx1G

-XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+ParallelRefProcEnabled 
-XX:MaxGCPauseMillis=200

I got them from this list so can’t take credit for them but they work for me.


Cheers

François


 On Feb 24, 2015, at 7:45 PM, Tang, Rebecca rebecca.t...@ucsf.edu wrote:
 
 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.
 
 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?
 
 
 Rebecca Tang
 Applications Developer, UCSF CKM
 Industry Documents Digital Libraries
 E: rebecca.t...@ucsf.edu
 
 
 
 
 
 On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:
 
 On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
 between 0-3 seconds to complete any query), but today I noticed that the
 performance is very bad (queries take between 12 ­ 15 seconds).
 
 I haven't updated the solr index configuration
 (schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
 every month, I rebuild the solr index from scratch and deploy it to the
 box.  We will eventually go to incremental builds. But for now, all
 indexes are built from scratch.
 
 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus
 
 The bottom line on this problem, and I'm sure it's not something you're
 going to want to hear:  You don't have enough memory available to cache
 your index.  I'd plan on at least 192GB of RAM for an index this size,
 and 256GB would be better.
 
 Depending on the exact index schema, the nature of your queries, and how
 large your Java heap for Solr is, 100GB of RAM could be enough for good
 performance on an index that size ... or it might be nowhere near
 enough.  I would imagine that one of two things is true here, possibly
 both:  1) Your queries are very complex and involve accessing a very
 large percentage of the index data.  2) Your Java heap is enormous,
 leaving very little RAM for the OS to automatically cache the index.
 
 Adding more memory to the machine, if that's possible, might fix some of
 the problems.  You can find a discussion of the problem here:
 
 http://wiki.apache.org/solr/SolrPerformanceProblems
 
 If you have any questions after reading that wiki article, feel free to
 ask them.
 
 Thanks,
 Shawn
 
 

Re: how to debug solr performance degradation

[Boogie Shafer]

rebecca,

i would suggest making sure you have some gc logging configured so you have 
some visibility into the JVM, esp if you don't already have JMX for sflow agent 
configured to give you external visibility of those internal metrics

the options below just print out the gc activity to a log

-Xloggc:gc.log
-verbose:gc 
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+PrintClassHistogram 
-XX:+PrintHeapAtGC 
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure 
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintTLAB
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=5
-XX:GCLogFileSize=10m




on the memory tuning side if things, as has already been mentioned, try to 
leave as much memory (outside the JVM) available to your OS to cache as much of 
the actual index as possible

in your case, you have a lot of RAM, so i would suggest starting with the gc 
logging options above, plus these very basic JVM memory settings
-XX:+UseG1GC
-Xms2G
-Xmx4G
-XX:+UseAdaptiveSizePolicy 
-XX:MaxGCPauseMillis=1000 
-XX:GCTimeRatio=19

in short, start by letting the JVM tune itself ;)

then start looking at the actual GC behavior (this will be visible in the gc 
logs)


---
on the OS performance monitoring, a few real time tools which i like to use on 
linux

nmon
dstat
htop

for trending start with the basics (sysstat/sar) 
and build from there (hsflowd is super easy to install and get pushing data up 
to a central console like ganglia)
you can add to that by adding the sflow JVM agent to your solr environment

enabling JMX interface on jetty will let you use tools like jconsole or 
jvisualvm





From: François Schiettecatte fschietteca...@gmail.com
Sent: Tuesday, February 24, 2015 17:06
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

Rebecca

You don’t want to give all the memory to the JVM. You want to give it just 
enough for it to work optimally and leave the rest of the memory for the OS to 
use for caching data. Giving the JVM too much memory can result in worse 
performance because of GC. There is no magic formula to figuring out the memory 
allocation for the JVM, that is very dependent on the workload. In your case I 
would start with 5GB, and increment by 5GB with each run.

I also use these settings for the JVM

-XX:+UseG1GC -Xms1G -Xmx1G

-XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+ParallelRefProcEnabled 
-XX:MaxGCPauseMillis=200

I got them from this list so can’t take credit for them but they work for me.


Cheers

François


 On Feb 24, 2015, at 7:45 PM, Tang, Rebecca rebecca.t...@ucsf.edu wrote:

 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?


 Rebecca Tang
 Applications Developer, UCSF CKM
 Industry Documents Digital Libraries
 E: rebecca.t...@ucsf.edu





 On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
 between 0-3 seconds to complete any query), but today I noticed that the
 performance is very bad (queries take between 12 ­ 15 seconds).

 I haven't updated the solr index configuration
 (schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
 every month, I rebuild the solr index from scratch and deploy it to the
 box.  We will eventually go to incremental builds. But for now, all
 indexes are built from scratch.

 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

 The bottom line on this problem, and I'm sure it's not something you're
 going to want to hear:  You don't have enough memory available to cache
 your index.  I'd plan on at least 192GB of RAM for an index this size,
 and 256GB would be better.

 Depending on the exact index schema, the nature of your queries, and how
 large your Java heap for Solr is, 100GB of RAM could be enough for good
 performance on an index that size ... or it might be nowhere near
 enough.  I would imagine that one of two things is true here, possibly
 both:  1) Your queries are very complex and involve accessing a very
 large percentage of the index data.  2) Your Java heap is enormous,
 leaving very little RAM for the OS to automatically cache the index.

 Adding more memory to the machine, if that's possible, might fix some of
 the problems.  You can find a discussion of the problem here:

 http://wiki.apache.org/solr/SolrPerformanceProblems

 If you have any questions after reading that wiki article, feel free to
 ask them.

 Thanks,
 Shawn

Re: how to debug solr performance degradation

[Boogie Shafer]

meant to type JMX or sflow agent

also should have mentioned you want to be running a very recent JDK


From: Boogie Shafer boogie.sha...@proquest.com
Sent: Tuesday, February 24, 2015 18:03
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

rebecca,

i would suggest making sure you have some gc logging configured so you have 
some visibility into the JVM, esp if you don't already have JMX for sflow agent 
configured to give you external visibility of those internal metrics

the options below just print out the gc activity to a log

-Xloggc:gc.log
-verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+PrintClassHistogram
-XX:+PrintHeapAtGC
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintTLAB
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=5
-XX:GCLogFileSize=10m




on the memory tuning side if things, as has already been mentioned, try to 
leave as much memory (outside the JVM) available to your OS to cache as much of 
the actual index as possible

in your case, you have a lot of RAM, so i would suggest starting with the gc 
logging options above, plus these very basic JVM memory settings
-XX:+UseG1GC
-Xms2G
-Xmx4G
-XX:+UseAdaptiveSizePolicy
-XX:MaxGCPauseMillis=1000
-XX:GCTimeRatio=19

in short, start by letting the JVM tune itself ;)

then start looking at the actual GC behavior (this will be visible in the gc 
logs)


---
on the OS performance monitoring, a few real time tools which i like to use on 
linux

nmon
dstat
htop

for trending start with the basics (sysstat/sar)
and build from there (hsflowd is super easy to install and get pushing data up 
to a central console like ganglia)
you can add to that by adding the sflow JVM agent to your solr environment

enabling JMX interface on jetty will let you use tools like jconsole or 
jvisualvm





From: François Schiettecatte fschietteca...@gmail.com
Sent: Tuesday, February 24, 2015 17:06
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

Rebecca

You don’t want to give all the memory to the JVM. You want to give it just 
enough for it to work optimally and leave the rest of the memory for the OS to 
use for caching data. Giving the JVM too much memory can result in worse 
performance because of GC. There is no magic formula to figuring out the memory 
allocation for the JVM, that is very dependent on the workload. In your case I 
would start with 5GB, and increment by 5GB with each run.

I also use these settings for the JVM

-XX:+UseG1GC -Xms1G -Xmx1G

-XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+ParallelRefProcEnabled 
-XX:MaxGCPauseMillis=200

I got them from this list so can’t take credit for them but they work for me.


Cheers

François


 On Feb 24, 2015, at 7:45 PM, Tang, Rebecca rebecca.t...@ucsf.edu wrote:

 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.

 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?


 Rebecca Tang
 Applications Developer, UCSF CKM
 Industry Documents Digital Libraries
 E: rebecca.t...@ucsf.edu





 On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
 between 0-3 seconds to complete any query), but today I noticed that the
 performance is very bad (queries take between 12 ­ 15 seconds).

 I haven't updated the solr index configuration
 (schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
 every month, I rebuild the solr index from scratch and deploy it to the
 box.  We will eventually go to incremental builds. But for now, all
 indexes are built from scratch.

 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

 The bottom line on this problem, and I'm sure it's not something you're
 going to want to hear:  You don't have enough memory available to cache
 your index.  I'd plan on at least 192GB of RAM for an index this size,
 and 256GB would be better.

 Depending on the exact index schema, the nature of your queries, and how
 large your Java heap for Solr is, 100GB of RAM could be enough for good
 performance on an index that size ... or it might be nowhere near
 enough.  I would imagine that one of two things is true here, possibly
 both:  1) Your queries are very complex and involve accessing a very
 large percentage of the index data.  2) Your Java heap is enormous,
 leaving very little RAM for the OS to automatically cache the index.

 Adding more memory to the machine

Re: how to debug solr performance degradation

[Walter Underwood]

The other memory is used by the OS as file buffers. All the important parts of 
the on-disk search index are buffered in memory. When the Solr process wants a 
block, it is already right there, no delays for disk access.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


On Feb 24, 2015, at 4:45 PM, Tang, Rebecca rebecca.t...@ucsf.edu wrote:

 We gave the machine 180G mem to see if it improves performance.  However,
 after we increased the memory, Solr started using only 5% of the physical
 memory.  It has always used 90-something%.
 
 What could be causing solr to not grab all the physical memory (grabbing
 so little of the physical memory)?
 
 Rebecca Tang
 Applications Developer, UCSF CKM
 Industry Documents Digital Libraries
 E: rebecca.t...@ucsf.edu
 
 On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:
 
 On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
 between 0-3 seconds to complete any query), but today I noticed that the
 performance is very bad (queries take between 12 ­ 15 seconds).
 
 I haven't updated the solr index configuration
 (schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
 every month, I rebuild the solr index from scratch and deploy it to the
 box.  We will eventually go to incremental builds. But for now, all
 indexes are built from scratch.
 
 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus
 
 The bottom line on this problem, and I'm sure it's not something you're
 going to want to hear:  You don't have enough memory available to cache
 your index.  I'd plan on at least 192GB of RAM for an index this size,
 and 256GB would be better.
 
 Depending on the exact index schema, the nature of your queries, and how
 large your Java heap for Solr is, 100GB of RAM could be enough for good
 performance on an index that size ... or it might be nowhere near
 enough.  I would imagine that one of two things is true here, possibly
 both:  1) Your queries are very complex and involve accessing a very
 large percentage of the index data.  2) Your Java heap is enormous,
 leaving very little RAM for the OS to automatically cache the index.
 
 Adding more memory to the machine, if that's possible, might fix some of
 the problems.  You can find a discussion of the problem here:
 
 http://wiki.apache.org/solr/SolrPerformanceProblems
 
 If you have any questions after reading that wiki article, feel free to
 ask them.
 
 Thanks,
 Shawn
 
 

Re: how to debug solr performance degradation

[Tang, Rebecca]

We gave the machine 180G mem to see if it improves performance.  However,
after we increased the memory, Solr started using only 5% of the physical
memory.  It has always used 90-something%.

What could be causing solr to not grab all the physical memory (grabbing
so little of the physical memory)?


Rebecca Tang
Applications Developer, UCSF CKM
Industry Documents Digital Libraries
E: rebecca.t...@ucsf.edu





On 2/24/15 12:44 PM, Shawn Heisey apa...@elyograg.org wrote:

On 2/24/2015 1:09 PM, Tang, Rebecca wrote:
 Our solr index used to perform OK on our beta production box (anywhere
between 0-3 seconds to complete any query), but today I noticed that the
performance is very bad (queries take between 12 ­ 15 seconds).

 I haven't updated the solr index configuration
(schema.xml/solrconfig.xml) lately.  All that's changed is the data ‹
every month, I rebuild the solr index from scratch and deploy it to the
box.  We will eventually go to incremental builds. But for now, all
indexes are built from scratch.

 Here are the stats:
 Solr index size 183G
 Documents in index 14364201
 We just have single solr box
 It has 100G memory
 500G Harddrive
 16 cpus

The bottom line on this problem, and I'm sure it's not something you're
going to want to hear:  You don't have enough memory available to cache
your index.  I'd plan on at least 192GB of RAM for an index this size,
and 256GB would be better.

Depending on the exact index schema, the nature of your queries, and how
large your Java heap for Solr is, 100GB of RAM could be enough for good
performance on an index that size ... or it might be nowhere near
enough.  I would imagine that one of two things is true here, possibly
both:  1) Your queries are very complex and involve accessing a very
large percentage of the index data.  2) Your Java heap is enormous,
leaving very little RAM for the OS to automatically cache the index.

Adding more memory to the machine, if that's possible, might fix some of
the problems.  You can find a discussion of the problem here:

http://wiki.apache.org/solr/SolrPerformanceProblems

If you have any questions after reading that wiki article, feel free to
ask them.

Thanks,
Shawn

Re: how to debug dataimporthandler

[PeterKerk]

Tried your steps, but failed. Could you perhaps have a look at my post here:
http://lucene.472066.n3.nabble.com/org-apache-solr-handler-dataimport-DataImportHandlerException-Unable-to-execute-query-td4108227.html



--
View this message in context: 
http://lucene.472066.n3.nabble.com/how-to-debug-dataimporthandler-tp2611506p4108676.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to debug my own analyzer in solr

[Mingzhu Gao]

More information about this , the custom analyzer just implement
createComponents of Analyzer.

And my configure in schema.xml is just something like :

fieldType name=text_cn class=solr.TextField 
 analyzer class=my.package.CustomAnalyzer /
/fieldType


From the log I cannot see any error information , however , when I want to
analysis or add document data , it always hang there .

Any way to debug or narrow down the problem ?

Thanks in advance .

-Mingz

On 10/21/13 4:35 PM, Mingzhu Gao m...@adobe.com wrote:

Dear solr expert ,

I would like to write my own analyser ( Chinese analyser ) and integrate
them into solr as solr plugin .

From the log information , the custom analyzer can be loaded into solr
successfully .  I define my fieldType with this custom analyzer.

Now the problem is that ,  when I try this analyzer from
http://localhost:8983/solr/#/collection1/analysis , click the analysis ,
then choose my FieldType , then input some text .
After I click Analyse Value button , the solr hang there , I cannot get
any result or response in a few minutes.

I also try to add  some data by curl
http://localhost:8983/solr/update?commit=true -H Content-Type: text/xml
, or by post.sh in exampledocs folder ,
The same issue , the solr hang there , no result and not response .

Can anybody give me some suggestions on how to debug solr to work with my
own custom analyzer ?

By the way , I write a java program to call my custom analyzer , the
result is okay , for example , the following code can work well .
==
Analyzer analyzer = new MyAnalyzer() ;

TokenStream ts = analyzer.tokenStream() ;

CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);

ts.reset();

while (ts.incrementToken()){

System.out.println(ta.toString());

}

=


Thanks,

-Mingz

Re: how to debug my own analyzer in solr

[Siegfried Goeschl]

Thread Dump and/or Remote Debugging?!

Cheers,

Siegfried Goeschl

On 21.10.13 11:58, Mingzhu Gao wrote:

More information about this , the custom analyzer just implement
createComponents of Analyzer.

And my configure in schema.xml is just something like :

fieldType name=text_cn class=solr.TextField 
  analyzer class=my.package.CustomAnalyzer /
/fieldType


 From the log I cannot see any error information , however , when I want to
analysis or add document data , it always hang there .

Any way to debug or narrow down the problem ?

Thanks in advance .

-Mingz

On 10/21/13 4:35 PM, Mingzhu Gao m...@adobe.com wrote:


Dear solr expert ,

I would like to write my own analyser ( Chinese analyser ) and integrate
them into solr as solr plugin .

From the log information , the custom analyzer can be loaded into solr
successfully .  I define my fieldType with this custom analyzer.

Now the problem is that ,  when I try this analyzer from
http://localhost:8983/solr/#/collection1/analysis , click the analysis ,
then choose my FieldType , then input some text .
After I click Analyse Value button , the solr hang there , I cannot get
any result or response in a few minutes.

I also try to add  some data by curl
http://localhost:8983/solr/update?commit=true -H Content-Type: text/xml
, or by post.sh in exampledocs folder ,
The same issue , the solr hang there , no result and not response .

Can anybody give me some suggestions on how to debug solr to work with my
own custom analyzer ?

By the way , I write a java program to call my custom analyzer , the
result is okay , for example , the following code can work well .
==
Analyzer analyzer = new MyAnalyzer() ;

TokenStream ts = analyzer.tokenStream() ;

CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);

ts.reset();

while (ts.incrementToken()){

System.out.println(ta.toString());

}

=


Thanks,

-Mingz

Re: how to debug my own analyzer in solr

[Koji Sekiguchi]

Hi Mingz,

If you use Eclipse, you can debug Solr with your plugin like this:

# go to Solr install directory
$ cd $SOLR
$ ant run-example -Dexample.debug=true

Then connect the JVM from Eclipse via remote debug port 5005.

Good luck!

koji


(13/10/21 18:58), Mingzhu Gao wrote:

More information about this , the custom analyzer just implement
createComponents of Analyzer.

And my configure in schema.xml is just something like :

fieldType name=text_cn class=solr.TextField 
  analyzer class=my.package.CustomAnalyzer /
/fieldType



From the log I cannot see any error information , however , when I want to

analysis or add document data , it always hang there .

Any way to debug or narrow down the problem ?

Thanks in advance .

-Mingz

On 10/21/13 4:35 PM, Mingzhu Gao m...@adobe.com wrote:


Dear solr expert ,

I would like to write my own analyser ( Chinese analyser ) and integrate
them into solr as solr plugin .

From the log information , the custom analyzer can be loaded into solr
successfully .  I define my fieldType with this custom analyzer.

Now the problem is that ,  when I try this analyzer from
http://localhost:8983/solr/#/collection1/analysis , click the analysis ,
then choose my FieldType , then input some text .
After I click Analyse Value button , the solr hang there , I cannot get
any result or response in a few minutes.

I also try to add  some data by curl
http://localhost:8983/solr/update?commit=true -H Content-Type: text/xml
, or by post.sh in exampledocs folder ,
The same issue , the solr hang there , no result and not response .

Can anybody give me some suggestions on how to debug solr to work with my
own custom analyzer ?

By the way , I write a java program to call my custom analyzer , the
result is okay , for example , the following code can work well .
==
Analyzer analyzer = new MyAnalyzer() ;

TokenStream ts = analyzer.tokenStream() ;

CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);

ts.reset();

while (ts.incrementToken()){

System.out.println(ta.toString());

}

=


Thanks,

-Mingz







--
http://soleami.com/blog/automatically-acquiring-synonym-knowledge-from-wikipedia.html

Re: How to debug an OutOfMemoryError?

[SolrLover]

Are you using softcommits heavily? I heard that using softcommits heavily
(every second) and not using hard commit for long time causes out of memory
issues since SOLR uses hashmap for transaction log.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-debug-an-OutOfMemoryError-tp4080085p4080090.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to debug an OutOfMemoryError?

[Roman Chyla]

_One_ idea would be to configure your java to dump core on the oom error -
you can then load the dump into some analyzers, eg. Eclipse, and that may
give you the desired answers (I fortunately don't remember that from top of
my head how to activate the dump, but google will give your the answer)

roman


On Wed, Jul 24, 2013 at 11:38 AM, jimtronic jimtro...@gmail.com wrote:

 I've encountered an OOM that seems to come after the server has been up
 for a
 few weeks.

 While I would love for someone to just tell me you did X wrong, I'm more
 interested in trying to debug this. So, given the error below, where would
 I
 look next? The only odd thing that sticks out to me is that my log file had
 grown to about 70G. Would that cause an error like this? This is Solr 4.2.

 Jul 24, 2013 3:08:09 PM org.apache.solr.common.SolrException log
 SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java
 heap space
 at

 org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:651)
 at

 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:364)
 at

 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
 at

 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
 at
 org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
 at

 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at
 org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
 at

 org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at

 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
 at
 org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
 at

 org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at

 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
 at

 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at

 org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
 at

 org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
 at

 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:365)
 at

 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
 at

 org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
 at

 org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
 at

 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:642)
 at
 org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
 at

 org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
 at

 org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
 at

 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at

 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
 at java.lang.Thread.run(Thread.java:619)
 Caused by: java.lang.OutOfMemoryError: Java heap space
 at org.apache.lucene.util.OpenBitSet.init(OpenBitSet.java:88)
 at
 org.apache.solr.search.DocSetCollector.collect(DocSetCollector.java:65)
 at org.apache.lucene.search.Scorer.score(Scorer.java:64)
 at
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:605)
 at
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
 at

 org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1060)
 at

 org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:763)
 at

 org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:880)
 at org.apache.solr.search.Grouping.execute(Grouping.java:284)
 at

 org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:384)
 at

 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
 at

 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1797)
 at

 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
 at

 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
 at

 

Re: How to debug an OutOfMemoryError?

[Shawn Heisey]

On 7/24/2013 9:38 AM, jimtronic wrote:
 I've encountered an OOM that seems to come after the server has been up for a
 few weeks. 
 
 While I would love for someone to just tell me you did X wrong, I'm more
 interested in trying to debug this. So, given the error below, where would I
 look next? The only odd thing that sticks out to me is that my log file had
 grown to about 70G. Would that cause an error like this? This is Solr 4.2.

Which log file is 70GB?  If this is the tlog directory, then SolrLover
is probably right - you need to do some hard commits, probably with
openSearcher set to false.

http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_startup

If it's not the tlog directory, then you might simply need to make your
heap larger, or look into ways to reduce heap usage.

https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
https://wiki.apache.org/solr/SolrPerformanceProblems#Reducing_heap_requirements

Thanks,
Shawn

Re: How to debug DIH with MySQL?

[dan whelan]

just a guess but this might need to change from

${biblio.id}

to

${book.id}

Since the entity name is book instead of biblio



On 1/10/12 10:37 AM, Walter Underwood wrote:

I see a missing required title field for every document when I'm using DIH. 
Yes, these documents have titles in the database. Is there a way to see what exact 
queries are sent to MySQL or received by MySQL?

Here is a relevant chunk of the dataConfig:

 entity name=book query=select * from biblio where idlt; 1
   field name=id column=id /
   field name=isbn column=isbn /
   field name=isbn column=ean /

   entity name=biblio_title
  query=select title from biblio_title where 
biblio_id='${biblio.id}'
field name=title column=title /
   /entity

wunder
--
Walter Underwood
wun...@wunderwood.org
Search Guy, Chegg

Re: How to debug DIH with MySQL?

[Walter Underwood]

Thanks! That looks like it fixed the problem. This list continues to be awesome.

Is the function of the name attribute actually described in the docs? I could 
not figure out what it was for.

wunder

On Jan 10, 2012, at 10:41 AM, dan whelan wrote:

 just a guess but this might need to change from
 
 ${biblio.id}
 
 to
 
 ${book.id}
 
 Since the entity name is book instead of biblio
 
 
 
 On 1/10/12 10:37 AM, Walter Underwood wrote:
 I see a missing required title field for every document when I'm using 
 DIH. Yes, these documents have titles in the database. Is there a way to see 
 what exact queries are sent to MySQL or received by MySQL?
 
 Here is a relevant chunk of the dataConfig:
 
 entity name=book query=select * from biblio where idlt; 1
   field name=id column=id /
   field name=isbn column=isbn /
   field name=isbn column=ean /
 
   entity name=biblio_title
query=select title from biblio_title where 
 biblio_id='${biblio.id}'
  field name=title column=title /
   /entity
 
 wunder
 --
 Walter Underwood
 wun...@wunderwood.org
 Search Guy, Chegg
 
 

--
Walter Underwood
wun...@wunderwood.org

Re: How to debug DIH with MySQL?

[Gora Mohanty]

On Wed, Jan 11, 2012 at 12:37 AM, Walter Underwood
wun...@wunderwood.org wrote:
 Thanks! That looks like it fixed the problem. This list continues to be 
 awesome.

 Is the function of the name attribute actually described in the docs? I could 
 not figure out what it was for.

Yes, it is, though maybe not very prominently.

From under http://wiki.apache.org/solr/DataImportHandler,
Configuration in data-config.xml  Schema for the data config :

The default attributes for an entity are:

* name (required) : A unique name used to identify an entity
...

Regards,
Gora

Re: How to debug DIH with MySQL?

[Walter Underwood]

Right, but that says exactly nothing about how that identifier is used. --wunder

On Jan 10, 2012, at 11:23 AM, Gora Mohanty wrote:

 On Wed, Jan 11, 2012 at 12:37 AM, Walter Underwood
 wun...@wunderwood.org wrote:
 Thanks! That looks like it fixed the problem. This list continues to be 
 awesome.
 
 Is the function of the name attribute actually described in the docs? I 
 could not figure out what it was for.
 
 Yes, it is, though maybe not very prominently.
 
 From under http://wiki.apache.org/solr/DataImportHandler,
 Configuration in data-config.xml  Schema for the data config :
 
 The default attributes for an entity are:
 
* name (required) : A unique name used to identify an entity
 ...
 
 Regards,
 Gora

Re: How to debug if termsComponent is used

[cyang2010]

I tried it.  It just does not work.   the debug component only works when
query component is there, and it is just showing debugging information for
query result, not term match result.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-debug-if-termsComponent-is-used-tp2891735p2895647.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to debug if termsComponent is used

[Erick Erickson]

Saying it does not work doesn't give us much to go on. Can you describe
what you've tried? *How* it fails? Have you looked in the log for any clues?

You might review this page:
http://wiki.apache.org/solr/UsingMailingLists

Best
Erick

On Tue, May 3, 2011 at 3:35 PM, cyang2010 ysxsu...@hotmail.com wrote:
 I tried it.  It just does not work.   the debug component only works when
 query component is there, and it is just showing debugging information for
 query result, not term match result.

 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/How-to-debug-if-termsComponent-is-used-tp2891735p2895647.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to debug if termsComponent is used

[cyang2010]

Sorry i didn't mean to give random reply.  It is just today my solr
configuration/schema is different now and can't get the error message.

Anyway, i rerun the test.

Basically by specifying such searchcomponent and requesthandler, you won't
get any error.

When you query this it is fine without any debug message (of course, since
no debug parameter is defined in query).

http://localhost:8080/solr/titles/terms?terms=trueterms.fl=autosuggestterms.prefix=andyterms.mincount=1;

As soon as i specify the only debug parameter i know, debugQuery the solr
server give this error:
http://localhost:8080/solr/titles/terms?terms=trueterms.fl=autosuggestterms.prefix=andyterms.mincount=1debugQuery=true


May 3, 2011 1:27:37 PM org.apache.solr.core.SolrCore execute
INFO: [titles] webapp=/solr path=/terms
params={debugQuery=trueterms.mincount=1
terms.fl=autosuggestterms=trueterms.prefix=andy} status=500 QTime=641
May 3, 2011 1:27:37 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
at
org.apache.solr.handler.component.DebugComponent.process(DebugCompone
nt.java:54)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sea
rchHandler.java:203)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl
erBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter
.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte
r.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
568)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:44
7)
at java.lang.Thread.run(Thread.java:619)



That is all i get.  Let me know if i use the wrong parameter or what.

Thanks.


cy

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-debug-if-termsComponent-is-used-tp2891735p2895897.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to debug if termsComponent is used

[Otis Gospodnetic]

Hi,

That looks about right, but I don't know without checking around if debug 
component really needs query component, or if it can work with just terms 
component.
Have you tried it?  Did it not work?

You may save yourself a lot of work and get something better than terms 
component with http://sematext.com/products/autocomplete/index.html btw.  Or if 
you are using Solr trunk, with Suggester.

Otis

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



- Original Message 
 From: cyang2010 ysxsu...@hotmail.com
 To: solr-user@lucene.apache.org
 Sent: Mon, May 2, 2011 6:57:49 PM
 Subject: How to debug if termsComponent is used
 
 Hi, I defined a searchHanlder just for the sake of autosuggest,  using
 TermsComponent.
 
   searchComponent  name=terms
 class=org.apache.solr.handler.component.TermsComponent  
   /searchComponent
 
 
   requestHandler  name=/terms
 class=org.apache.solr.handler.component.SearchHandler
  lst name=defaults
   str  name=echoParamsexplicit/str
  /lst
 
 arr name=components
strterms/str
strdebug/str
 /arr
   
 
 This configuration might not even make sense, to configure terms  and
 debug component together.  Is debug component must be wired up  with
 query component?   I just need a requestHanlder where i can  run
 termsComponent, and debug on it.  How do I achieve  that?
 
 Thanks,
 
 cy
/requestHandler
 
 --
 View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-debug-if-termsComponent-is-used-tp2891735p2891735.html

 Sent  from the Solr - User mailing list archive at Nabble.com.
 

Re: how to debug dataimporthandler

[Stefan Matheis]

Hey,

normally .. if i have problems with dih:

* i start having a look at the mysql-query-log, to check which queries
are executed.
* re-run the query myself, verify the return data
* Activate http://wiki.apache.org/solr/DataImportHandler#LogTransformer
and log the important data, check console output

yet, this was debugging enough for all my problems :)

Regards
Stefan

On Wed, Mar 2, 2011 at 7:28 AM, cyang2010 ysxsu...@hotmail.com wrote:
 I wonder how to run dataimporthandler in debug mode.  Currently i can't get
 data correctly into index through dataimporthandler, especially a timestamp
 column to solr date field.  I want to debug the process.

 According to this wiki page:

 Commands
 The handler exposes all its API as http requests . The following are the
 possible operations
 •full-import : Full Import operation can be started by hitting the URL
 http://:/solr/dataimport?command=full-import
 ...
 ■clean : (default 'true'). Tells whether to clean up the index before the
 indexing is started
 ■commit: (default 'true'). Tells whether to commit after the operation
 ■optimize: (default 'true'). Tells whether to optimize after the operation
 ■debug : (default false). Runs in debug mode.It is used by the interactive
 development mode (see here)
 ■Please note that in debug mode, documents are never committed
 automatically. If you want to run debug mode and commit the results too, add
 'commit=true' as a request parameter.


 Therefore, i run

 http://:/solr/dataimport?command=full-import debug=true

 Not only i didn't see log with DEBUG level, but also it crashes my machine
 a few times.   I was surprised it can even do that ...

 Did someone ever try to debug the process before?  What is your experience
 with it?



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/how-to-debug-dataimporthandler-tp2611506p2611506.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

can nobody help me or want :D
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1288705.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[Bernd Fehling]

 
 can nobody help me or want :D

As already someone said:
- install Eclipse
- add Jetty Webapp Plugin to Eclipse
- add svn plugin to Eclipse
- download with svn the repository from trunk
- change to lucene dir and run ant package
- change to solr dir and run ant dist
- setup with Run configure... a Jetty Webapp for solr
- start debugging :-)

If debugging below solr level into lucene level just add
lucene src path to debugging source.

May be you should read:
http://www.lucidimagination.com/developers/articles/setting-up-apache-solr-in-eclipse

Regards,
Bernd

Re: How to Debug Sol-Code in Eclipse ?!

[Drew Farris]

On Sun, Aug 22, 2010 at 8:29 PM, stockii st...@shopgate.com wrote:

 okay, thx. but it want work =(

 i checkout solr1.4.1 as dynamic web project into eclipse. startet jetty with
 XDebug. In eclpise i add WebLogic exactly how the tutorial shows but eclipse
 cannot connect =(

 any idea what im doing wrong ?

No idea. Check your arguments and verify that the port is the same on
both the command-line you're using to start jetty and in the eclipse
remote debugger configuration. Make sure you don't have a firewall
running on your machine because that might block the connection from
eclipse to jetty.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

ant package  

BUILD FAILED run program perl ...

it`s necessary to install perl on my computer ?!

-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1291992.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

thx, for your help. now it works fine. its very simple when you kno how :D
haha

i try bernds suggest =) 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1296175.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

thx for you reply.

i dont want to test my own classes in unittest. i try to understand how solr
works , because i write a little text about solr and lucene. so i want go
through the code, step by step and find out on which places is solr using
lucene. 

when i can debug the code its easyer ;-)
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1274285.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[Geert-Jan Brits]

1. download solr lib and import them in your project.
2. download solr source-code of the same version and attach in to the
libraries. (I haven't got eclipse open but it is something like project -
settings - jre/libraries?)
3. write a small program yourself which calls EmbededSolrServer and
step-through/debug the source-code from there. It works just like it is your
own source-code.

HTH,
Geert-Jan

2010/8/22 stockii st...@shopgate.com


 thx for you reply.

 i dont want to test my own classes in unittest. i try to understand how
 solr
 works , because i write a little text about solr and lucene. so i want go
 through the code, step by step and find out on which places is solr using
 lucene.

 when i can debug the code its easyer ;-)
 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1274285.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

1. did you mean solr war ? or all libs in /trunk/solr/lib/
2. what is the revision of the actual 1.4.1 ?
3. EmbededSolrServer ? what should do this little programm ? 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1275295.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[Drew Farris]

On Sun, Aug 22, 2010 at 8:22 AM, stockii st...@shopgate.com wrote:

 thx for you reply.

 i dont want to test my own classes in unittest. i try to understand how solr
 works , because i write a little text about solr and lucene. so i want go
 through the code, step by step and find out on which places is solr using
 lucene.

 when i can debug the code its easyer ;-)

How about using remote debugging?

I assume you have the solr/lucene code set up in a project in eclipse.

Start solr with:
java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=1044
-jar start.jar

And then follow the instructions to attach to the debugger described here:
http://www.eclipsezone.com/eclipse/forums/t53459.html
(The instructions are for weblogic, but everything in the section
titled The Eclipse Connection is generally relevant.)

Drew

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

oha, thx i will try it =) 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1278704.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

wich revision is the revision of 1.4.1 ? 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1280956.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

im stupid. XD 

how get i the revsion of the solr 1.4.1 or do i nee build an war from my
snv-checkout ? 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1281253.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[Drew Farris]

After a build of what you have checked out, there's an instance you
can run in the examples directory within the solr directory.

1.4.1 can be checked out from
http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1/

On Sun, Aug 22, 2010 at 6:01 PM, stockii st...@shopgate.com wrote:

 im stupid. XD

 how get i the revsion of the solr 1.4.1 or do i nee build an war from my
 snv-checkout ?
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1281253.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[stockii]

okay, thx. but it want work =(

i checkout solr1.4.1 as dynamic web project into eclipse. startet jetty with
XDebug. In eclpise i add WebLogic exactly how the tutorial shows but eclipse
cannot connect =(

any idea what im doing wrong ? 
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1282560.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Debug Sol-Code in Eclipse ?!

[Lance Norskog]

Running unit tests is easy, once you set the right 'current directory'
so that unit tests can find their resource files. I have found that if
I get a full set of unit tests for something, I don't have to debug it
in the full app.

Running the whole thing as a servlet has the whole servlet engine
setup thing, which I avoid.

Running as EmbeddedSolr might be easy, I haven't tried it.

I usually make a separate empty Java project and import source and
libs as needed. I do a lot of  'search everything for this string' so
having the whole source tree in the project just slows me down. This
does remove the ability to use the svn/git management, but I don't
mind that.

On Sat, Aug 21, 2010 at 5:27 AM, stockii st...@shopgate.com wrote:

 Hello..

 Can anyone give me some tipps to debug the solr-code in Eclipse ? or do i
 need apache-Ant to do this ?

 thhx =)
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/How-to-Debug-Sol-Code-in-Eclipse-tp1262050p1262050.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Lance Norskog
goks...@gmail.com

Re: How to debug ?

[Brian Carmalt]

Hello Beto,

There is a plugin for jetty: http://webtide.com/eclipse. Insert this as
and update site and let eclipse install the plugin for you You can then
start the jetty server from eclipse and debug it. 

Brian. 

Am Mittwoch, den 25.06.2008, 12:48 +1000 schrieb Norberto Meijome:
 On Tue, 24 Jun 2008 19:17:58 -0700
 Ryan McKinley [EMAIL PROTECTED] wrote:
 
  also, check the LukeRequestHandler
  
  if there is a document you think *should* match, you can see what  
  tokens it has actually indexed...
 
 right, I will look into that a bit more. 
 
 I am actually using the lukeall.jar (0.8.1, linked against lucene 2.4) to look
 into what got indexed, but I am bit wary of how what I select in the the
 'analyzer' drop down option in Luke actually affects what I see.
 
 B
 
 _
 {Beto|Norberto|Numard} Meijome
 
 Web2.0 is outsourced RD from Web1.0 companies.
The Reverend
 
 I speak for myself, not my employer. Contents may be hot. Slippery when wet.
 Reading disclaimers makes you go blind. Writing them is worse. You have been
 Warned.

Re: How to debug ?

[Norberto Meijome]

On Wed, 25 Jun 2008 08:37:35 +0200
Brian Carmalt [EMAIL PROTECTED] wrote:

 There is a plugin for jetty: http://webtide.com/eclipse. Insert this as
 and update site and let eclipse install the plugin for you You can then
 start the jetty server from eclipse and debug it. 

Thanks Brian, good information :)

B

_
{Beto|Norberto|Numard} Meijome

Q. How do you make God laugh?
A. Tell him your plans.

I speak for myself, not my employer. Contents may be hot. Slippery when wet. 
Reading disclaimers makes you go blind. Writing them is worse. You have been 
Warned.

Re: How to debug ?

[Norberto Meijome]

On Tue, 24 Jun 2008 19:17:58 -0700
Ryan McKinley [EMAIL PROTECTED] wrote:

 also, check the LukeRequestHandler
 
 if there is a document you think *should* match, you can see what  
 tokens it has actually indexed...
 

hi Ryan,
I can't see the tokens generated using LukeRequestHandler.

I can get to the document I want : 
http://localhost:8983/solr/_test_/admin/luke/?id=Jay%20Rock

and for the field I am interested , i get only :
[...]
lst name=artist_ngram
str name=typengram/str
str name=schemaITS--/str
str name=flagsITS--/str
str name=valueJay Rock/str
str name=internalJay Rock/str
float name=boost1.0/float
int name=docFreq0/int
/lst
[...]

( all the other fields look pretty much identical , none of them show the 
tokens generated).

using the luke tool itself ( lukeall.jar ,source # 0.8.1, linked against 
Lucene's 2.4 libs bundled with the nightly build), I see the following tokens, 
for this document + field:

ja, ay, y ,  r, ro, 
oc, ck, jay, ay , y r, 
 ro, roc, ock, jay , ay r, 
y ro,  roc, rock, jay r, ay ro, 
y roc,  rock, jay ro, ay roc, y rock, 
jay roc, ay rock, jay rock

Which is precisely what I expect, given that my 'ngram' type is defined as :

!-- n-gram tokenization --
fieldType name=ngram class=solr.TextField
positionIncrementGap=100
analyzer type=index
tokenizer

class=org.apache.solr.analysis.NGramTokenizerFactory
minGramSize=2 maxGramSize=15 /
filter class=solr.LowerCaseFilterFactory /
filter 
class=solr.RemoveDuplicatesTokenFilterFactory /
/analyzer
analyzer type=query
tokenizer

class=org.apache.solr.analysis.NGramTokenizerFactory
minGramSize=2 maxGramSize=15 /
filter class=solr.LowerCaseFilterFactory /
filter 
class=solr.RemoveDuplicatesTokenFilterFactory /
/analyzer
/fieldType


My question now is, was I supposed to get any more information from 
LukeRequestHandler ?


furthermore, if I perform , on this same core with exactly this data :
http://localhost:8983/solr/_test_/select?q=artist_ngram:ro

I get this document returned (and many others).

but, if I search for 'roc' instead of 'ro' :
http://localhost:8983/solr/_test_/select?q=artist_ngram:roc

−
response
−
lst name=responseHeader
int name=status0/int
int name=QTime48/int
−
lst name=params
str name=qartist_ngram:roc/str
str name=debugQuerytrue/str
/lst
/lst
result name=response numFound=0 start=0/
−
lst name=debug
str name=rawquerystringartist_ngram:roc/str
str name=querystringartist_ngram:roc/str
str name=parsedqueryPhraseQuery(artist_ngram:ro oc roc)/str
str name=parsedquery_toStringartist_ngram:ro oc roc/str
lst name=explain/
str name=QParserOldLuceneQParser/str
−
lst name=timing
.[...]

Is searching on nGram tokenized fields  limited to the minGramSize ?

Thanks for any pointers you can provide,
B
_
{Beto|Norberto|Numard} Meijome

I didn't attend the funeral, but I sent a nice letter saying  I approved of 
it.
  Mark Twain

I speak for myself, not my employer. Contents may be hot. Slippery when wet. 
Reading disclaimers makes you go blind. Writing them is worse. You have been 
Warned.

Re: How to debug ?

[Ryan McKinley]

also, check the LukeRequestHandler

if there is a document you think *should* match, you can see what  
tokens it has actually indexed...



On Jun 24, 2008, at 7:12 PM, Norberto Meijome wrote:

hi,
I'm trying to understand why a search on a field tokenized with the  
nGram
tokenizer, with minGramSize=n and maxGramSize=m doesn't find any  
matches for

queries of length (in characters) of n+1..m (n works fine).

analysis.jsp shows that it SHOULD match, but /select doesn't bring  
anything
back. (For details on this queries, please see my previous post over  
the last

day or so to this list).

So i figure there is some difference between what analysis.jsp does  
and the
actual search executed , or what lucene indexes - i imagine  
analysis.jsp only
parses the input in the page with solr's tokenizers/filters but  
doesn't

actually do lucene's part of the job.

And I'd like to look into this... What is the suggested approach for  
this?
attach a debugger to jetty's web app ? Are there some pointers on  
how to debug

at this level? Preferably in Eclipse, but beggars cant be choosers ;)

thanks!!
B
_
{Beto|Norberto|Numard} Meijome

Always do right.  This will gratify some and astonish the rest.
 Mark Twain

I speak for myself, not my employer. Contents may be hot. Slippery  
when wet.
Reading disclaimers makes you go blind. Writing them is worse. You  
have been

Warned.

Re: How to debug ?

[Norberto Meijome]

On Tue, 24 Jun 2008 19:17:58 -0700
Ryan McKinley [EMAIL PROTECTED] wrote:

 also, check the LukeRequestHandler
 
 if there is a document you think *should* match, you can see what  
 tokens it has actually indexed...

right, I will look into that a bit more. 

I am actually using the lukeall.jar (0.8.1, linked against lucene 2.4) to look
into what got indexed, but I am bit wary of how what I select in the the
'analyzer' drop down option in Luke actually affects what I see.

B

_
{Beto|Norberto|Numard} Meijome

Web2.0 is outsourced RD from Web1.0 companies.
   The Reverend

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.