And think about distributed search, you are going through a 'proxy' which, as well as forwarding your docs, must also merge any docs from different shards into a single result set. That is likely to take some time on 30,000 docs, and isn't a job that is needed on non-distributed search.
Upayavira On Sun, Jan 20, 2013, at 08:55 PM, Walter Underwood wrote: > If you are going to request 30,000 rows, you can give up on getting good > performance. It is not going to happen. > > Even without all the disk accesses, think about how much is sent over the > network, then parsed by the client. The client cannot even start working > with the data until it is all received and parsed. > > wunder > > On Jan 20, 2013, at 8:49 AM, Michael Ryan wrote: > > > (This is based on my knowledge of 3.6 - not sure if this has changed in 4.0) > > > > You are using rows=30000, which requires retrieving 30000 documents from > > disk. In a non-distributed search, the QTime will not include the time it > > takes to retrieve these documents, but in a distributed search, it will. > > For a *:* query, the document retrieval will almost always be the slowest > > part of the query. I'd suggest measuring how long it takes for the response > > to be returned, or use rows=0. > > > > The timeAllowed feature is very misleading. It only applies to a small > > portion of the query (which in my experience is usually not the part of the > > query that is actually slow). Do not depend on timeAllowed doing anything > > useful :) > > > > -Michael > > > > -----Original Message----- > > From: Lyuba Romanchuk [mailto:lyuba.romanc...@gmail.com] > > Sent: Sunday, January 20, 2013 6:36 AM > > To: solr-user@lucene.apache.org > > Subject: Solr 4.0 - timeAllowed in distributed search > > > > Hi, > > > > I try to use timeAllowed in query both in distributed search with one shard > > and directly to the same shard. > > I send the same query with timeAllowed=500 : > > > > - directly to the shard then QTime ~= 600 ms > > - through distributes search to the same shard QTime ~= 7 sec. > > > > I have two questions: > > > > - It seems that timeAllowed parameter doesn't work for distributes > > search, does it? > > - What may be the reason that causes the query to the shard through > > distributes search takes much more time than to the shard directly (the > > same distribution remains without timeAllowed parameter in the query)? > > > > > > Test results: > > > > Ask one shard through distributed search: > > > > > > http://localhost:8983/solr/shard_2013-01-07/select?q=*:*&rows=30000&shards=127.0.0.1%3A8983%2Fsolr%2Fshard_2013-01-07&timeAllowed=500&partialResults=true&shards.info=true&debugQuery=true > > <response> > > <lst name="responseHeader"> > > <bool name="partialResults">true</bool> > > <int name="status">0</int> > > <int name="QTime">7307</int> > > <lst name="params"> > > <str name="q">*:*</str> > > <str name="shards">127.0.0.1:8983/solr/shard_2013-01-07</str> > > <str name="partialResults">true</str> > > <str name="debugQuery">true</str> > > <str name="shards.info">true</str> > > <str name="rows">30000</str> > > <str name="timeAllowed">500</str></lst></lst> > > <lst name="shards.info"> > > <lst name="127.0.0.1:8983/solr/shard_2013-01-07"> > > <long name="numFound">29574223</long> > > <float name="maxScore">1.0</float> > > <long name="time">646</long></lst></lst> > > <result name="response" numFound="29574223" start="0" maxScore="1.0"> ... > > 30,000 docs > > ... > > <lst name="debug"> > > <str name="rawquerystring">*:*</str> > > <str name="querystring">*:*</str> > > <str name="parsedquery">MatchAllDocsQuery(*:*)</str> > > <str name="parsedquery_toString">*:*</str> > > <str name="QParser">LuceneQParser</str> > > <lst name="timing"><double name="time">6141.0</double> <lst > > name="prepare"><double name="time">0.0</double> <lst > > name="org.apache.solr.handler.component.QueryComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.FacetComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.HighlightComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.StatsComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.DebugComponent"><double > > name="time">0.0</double></lst></lst> > > <lst name="process"><double name="time">6141.0</double> <lst > > name="org.apache.solr.handler.component.QueryComponent"><double > > name="time">6022.0</double></lst> > > <lst name="org.apache.solr.handler.component.FacetComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.HighlightComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.StatsComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.DebugComponent"><double > > name="time">119.0</double></lst></lst></lst>< > > > > Ask the same shard directly: > > > > http://localhost:8983/solr/shard_2013-01-07/select?q=*:*&rows=30000&timeAllowed=500&partialResults=true&shards.info=true&debugQuery=true > > <lst name="responseHeader"> > > <bool name="partialResults">true</bool> > > <int name="status">0</int> > > <int name="QTime">617</int> > > <lst name="params"> > > <str name="q">*:*</str> > > <str name="partialResults">true</str> > > <str name="debugQuery">true</str> > > <str name="shards.info">true</str> > > <str name="rows">30000</str> > > <str name="timeAllowed">500</str></lst></lst> > > <result name="response" numFound="28687243" start="0"> ... > > 30,000 docs > > <lst name="debug"><str name="rawquerystring">*:*</str><str > > name="querystring">*:*</str><str > > name="parsedquery">MatchAllDocsQuery(*:*)</str><str > > name="parsedquery_toString">*:*</str> > > <str name="QParser">LuceneQParser</str> > > <lst name="timing"><double name="time">617.0</double> <lst > > name="prepare"><double name="time">0.0</double> <lst > > name="org.apache.solr.handler.component.QueryComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.FacetComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.HighlightComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.StatsComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.DebugComponent"><double > > name="time">0.0</double></lst></lst> > > <lst name="process"><double name="time">617.0</double> <lst > > name="org.apache.solr.handler.component.QueryComponent"><double > > name="time">516.0</double></lst> > > <lst name="org.apache.solr.handler.component.FacetComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.MoreLikeThisComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.HighlightComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.StatsComponent"><double > > name="time">0.0</double></lst> > > <lst name="org.apache.solr.handler.component.DebugComponent"><double > > name="time">101.0</double></lst></lst></lst></lst> > > > > Thank you. > > Best regards, > > Lyuba > > -- > Walter Underwood > wun...@wunderwood.org > > >
