Performance Degradation on Migrating from 1.3 to solr 3.6.1
Hi,
On migrating from 1.3 to 3.6.1 , I see the query performance degrading by
nearly 2 times for all types of query. Indexing performance slight
degradation over 1.3 For Indexing we use our custom scripts that post xml
over HTTP.
Any thing that I might have missed . I am thinking that this might be due
to new Tiered MP over LogByteSize creating more segment files and hence
more the Query latency .We are using Compound Files in 1.3 and I have set
this to true even in 3.6.1 ,but results in more segement files
On optimizing the query response time improved beyond 1.3 .So could it be
the MP or am i missing something here . Do let me know
Please find attached the solrconfig.xml
Regards
Sujatha
?xml version=1.0 encoding=UTF-8 ?
!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the License); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an AS IS BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--
config
abortOnConfigurationError${solr.abortOnConfigurationError:true}/abortOnConfigurationError
luceneMatchVersionLUCENE_36/luceneMatchVersion
!-- The DirectoryFactory to use for indexes.
solr.StandardDirectoryFactory, the default, is filesystem based.
solr.RAMDirectoryFactory is memory based, not persistent, and doesn't work with replication. --
directoryFactory name=DirectoryFactory class=${solr.directoryFactory:solr.StandardDirectoryFactory}/
indexConfig
!-- Values here affect all index writers and act as a default unless overridden. --
useCompoundFiletrue/useCompoundFile
mergeFactor4/mergeFactor
maxFieldLength1/maxFieldLength
writeLockTimeout1000/writeLockTimeout
!-- mergePolicy class=org.apache.lucene.index.LogByteSizeMergePolicy / --
!-- mergePolicy class=org.apache.lucene.index.TieredMergePolicy
int name=maxMergeAtOnce4/int
int name=segmentsPerTier4/int
/mergePolicy --
lockTypesingle/lockType
/indexConfig
jmx /
updateHandler class=solr.DirectUpdateHandler2 /
query
maxBooleanClauses10/maxBooleanClauses
filterCache class=solr.FastLRUCache
size=16384
initialSize=4096
autowarmCount=1024 /
queryResultCache
class=solr.LRUCache
size=16384
initialSize=4096
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=16384
initialSize=4096
autowarmCount=0/
enableLazyFieldLoadingtrue/enableLazyFieldLoading
useFilterForSortedQuerytrue/useFilterForSortedQuery
queryResultWindowSize50/queryResultWindowSize
queryResultMaxDocsCached200/queryResultMaxDocsCached
useColdSearcherfalse/useColdSearcher
maxWarmingSearchers2/maxWarmingSearchers
/query
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false multipartUploadLimitInKB=2048 /
httpCaching never304=true /
!-- httpCaching lastModifiedFrom=openTime
etagSeed=Solr
/httpCaching --
/requestDispatcher
requestHandler name=standard class=solr.SearchHandler default=true
!-- default values for query parameters --
lst name=defaults
str name=echoParamsexplicit/str
!--
int name=rows10/int
str name=fl*/str
str name=version2.1/str
--
/lst
/requestHandler
requestHandler name=/update class=solr.XmlUpdateRequestHandler /
!-- requestHandler name=/analysis class=solr.AnalysisRequestHandler / --
requestHandler name=/admin/ class=solr.admin.AdminHandlers /
requestHandler name=/analysis/field
startup=lazy
class=solr.FieldAnalysisRequestHandler /
requestHandler name=/analysis/document
class=solr.DocumentAnalysisRequestHandler
startup=lazy /
requestHandler name=/admin/ping class=solr.PingRequestHandler
lst name=invariants
str name=qsolrpingquery/str
/lst
lst name=defaults
str name=echoParamsall/str
/lst
/requestHandler
!-- Echo the request contents back to the client --
requestHandler name=/debug/dump class=solr.DumpRequestHandler
lst name=defaults
str name=echoParamsexplicit/str !-- for all params (including the default etc) use: 'all' --
str
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Run a query on both old and new with debugQuery=true on your query request and look at the component timings for possible insight. -- Jack Krupansky From: Sujatha Arun Sent: Monday, September 24, 2012 7:26 AM To: solr-user@lucene.apache.org Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1 Hi, On migrating from 1.3 to 3.6.1 , I see the query performance degrading by nearly 2 times for all types of query. Indexing performance slight degradation over 1.3 For Indexing we use our custom scripts that post xml over HTTP. Any thing that I might have missed . I am thinking that this might be due to new Tiered MP over LogByteSize creating more segment files and hence more the Query latency .We are using Compound Files in 1.3 and I have set this to true even in 3.6.1 ,but results in more segement files On optimizing the query response time improved beyond 1.3 .So could it be the MP or am i missing something here . Do let me know Please find attached the solrconfig.xml Regards Sujatha
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Thanks Jack. so Qtime = Sum of all prepare components + sum of all process components - Debug comp process/prepare time In 3.6.1 the process part of Query component for the following query seems to take 8 times more time? anything missing? For most queries the process part of the Querycomponent seem to take more time in 3.6.1 This is *3.6.1 * response lst name=responseHeaderint name=status0/int *int name=QTime33/int* lst name=params str name=debugQueryon/str str name=indenton/strstr name=start0/str *str name=qdifferential AND equations AND has AND one AND solution/str * str name=rows10/strstr name=version2.2/str/lst/lst *Debug Output* * * str name=QParserLuceneQParser/str lst name=timing double name=time33.0/double lst name=prepare double name=time3.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst*lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double /lst/lstlst name=processdouble name=time30.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time26.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time4.0/double/lst *Same query in solr 1.3* * * lst name=responseHeader int name=status0/int *int name=QTime6/int* lst name=params str name=debugQueryon/strstr name=indenton/str str name=start0/strstr name=qdifferential AND equations AND has AND one AND solution/str str name=rows10/strstr name=version2.2/str Debug Info lst name=timing double name=time6.0/double lst name=prepare double name=time1.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time1.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double/lst/lst lst name=process double name=time5.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time2.0/double *On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky j...@basetechnology.comwrote: * Run a query on both old and new with debugQuery=true on your query request and look at the component timings for possible insight. -- Jack Krupansky From: Sujatha Arun Sent: Monday, September 24, 2012 7:26 AM To: solr-user@lucene.apache.org Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1 Hi, On migrating from 1.3 to 3.6.1 , I see the query performance degrading by nearly 2 times for all types of query. Indexing performance slight degradation over 1.3 For Indexing we use our custom scripts that post xml over HTTP. Any thing that I might have missed . I am thinking that this might be due to new Tiered MP over LogByteSize creating more segment files and hence more the Query latency .We are using Compound Files in 1.3 and I have set this to true even in 3.6.1 ,but results in more segement files On optimizing the query response time improved beyond 1.3 .So could it be the MP or am i missing something here . Do let me know Please find attached the solrconfig.xml Regards Sujatha
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Any comments on this? On Mon, Sep 24, 2012 at 10:28 PM, Sujatha Arun suja.a...@gmail.com wrote: Thanks Jack. so Qtime = Sum of all prepare components + sum of all process components - Debug comp process/prepare time In 3.6.1 the process part of Query component for the following query seems to take 8 times more time? anything missing? For most queries the process part of the Querycomponent seem to take more time in 3.6.1 This is *3.6.1 * response lst name=responseHeaderint name=status0/int *int name=QTime33/int* lst name=params str name=debugQueryon/str str name=indenton/strstr name=start0/str *str name=qdifferential AND equations AND has AND one AND solution/str* str name=rows10/strstr name=version2.2/str/lst/lst *Debug Output* * * str name=QParserLuceneQParser/str lst name=timing double name=time33.0/double lst name=prepare double name=time3.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst*lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double /lst/lstlst name=processdouble name=time30.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time26.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time4.0/double/lst *Same query in solr 1.3* * * lst name=responseHeader int name=status0/int *int name=QTime6/int* lst name=params str name=debugQueryon/strstr name=indenton/str str name=start0/strstr name=qdifferential AND equations AND has AND one AND solution/str str name=rows10/strstr name=version2.2/str Debug Info lst name=timing double name=time6.0/double lst name=prepare double name=time1.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time1.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double/lst/lst lst name=process double name=time5.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time2.0/double *On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky j...@basetechnology.comwrote: * Run a query on both old and new with debugQuery=true on your query request and look at the component timings for possible insight. -- Jack Krupansky From: Sujatha Arun Sent: Monday, September 24, 2012 7:26 AM To: solr-user@lucene.apache.org Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1 Hi, On migrating from 1.3 to 3.6.1 , I see the query performance degrading by nearly 2 times for all types of query. Indexing performance slight degradation over 1.3 For Indexing we use our custom scripts that post xml over HTTP. Any thing that I might have missed . I am thinking that this might be due to new Tiered MP over LogByteSize creating more segment files and hence more the Query latency .We are using Compound Files in 1.3 and I have set this to true even in 3.6.1 ,but results in more segement files On optimizing the query response time improved beyond 1.3 .So could it be the MP or am i missing something here . Do let me know Please find attached the solrconfig.xml Regards Sujatha
Re: Performance Degradation on Migrating from 1.3 to solr 3.6.1
Hi , Please comment on whether I should consider to move to the old Logbytesize MP on moving to 3.6.1 from 1.3 ,as I see improvements in query performance on optimization. Just to mention we have a lot of indexes in multi cores as well as multiple webapps and that's the reason we went for CFS in 1.3 to avoid the too many open file issue which we had encountered. Regards Sujatha On Tue, Sep 25, 2012 at 9:55 AM, Sujatha Arun suja.a...@gmail.com wrote: Any comments on this? On Mon, Sep 24, 2012 at 10:28 PM, Sujatha Arun suja.a...@gmail.comwrote: Thanks Jack. so Qtime = Sum of all prepare components + sum of all process components - Debug comp process/prepare time In 3.6.1 the process part of Query component for the following query seems to take 8 times more time? anything missing? For most queries the process part of the Querycomponent seem to take more time in 3.6.1 This is *3.6.1 * response lst name=responseHeaderint name=status0/int *int name=QTime33/int* lst name=params str name=debugQueryon/str str name=indenton/strstr name=start0/str *str name=qdifferential AND equations AND has AND one AND solution/str* str name=rows10/strstr name=version2.2/str/lst/lst *Debug Output* * * str name=QParserLuceneQParser/str lst name=timing double name=time33.0/double lst name=prepare double name=time3.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst*lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double /lst/lstlst name=processdouble name=time30.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time26.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.StatsComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time4.0/double/lst *Same query in solr 1.3* * * lst name=responseHeader int name=status0/int *int name=QTime6/int* lst name=params str name=debugQueryon/strstr name=indenton/str str name=start0/strstr name=qdifferential AND equations AND has AND one AND solution/str str name=rows10/strstr name=version2.2/str Debug Info lst name=timing double name=time6.0/double lst name=prepare double name=time1.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time1.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time0.0/double/lst/lst lst name=process double name=time5.0/double *lst name=org.apache.solr.handler.component.QueryComponentdouble name=time3.0/double/lst* lst name=org.apache.solr.handler.component.FacetComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.MoreLikeThisComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.HighlightComponentdouble name=time0.0/double/lstlst name=org.apache.solr.handler.component.DebugComponentdouble name=time2.0/double *On Mon, Sep 24, 2012 at 7:35 PM, Jack Krupansky j...@basetechnology.com wrote:* Run a query on both old and new with debugQuery=true on your query request and look at the component timings for possible insight. -- Jack Krupansky From: Sujatha Arun Sent: Monday, September 24, 2012 7:26 AM To: solr-user@lucene.apache.org Subject: Performance Degradation on Migrating from 1.3 to solr 3.6.1 Hi, On migrating from 1.3 to 3.6.1 , I see the query performance degrading by nearly 2 times for all types of query. Indexing performance slight degradation over 1.3 For Indexing we use our custom scripts that post xml over HTTP. Any thing that I might have missed . I am thinking that this might be due to new Tiered MP over LogByteSize creating more segment files and hence more the Query latency .We are using Compound Files in 1.3 and I have set this to true even in 3.6.1 ,but results in more segement files On optimizing the query response time improved beyond 1.3 .So could it be the MP or am i missing something here . Do let me know Please find attached the solrconfig.xml
