[
https://issues.apache.org/jira/browse/SOLR-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235224#comment-16235224
]
bidorbuy commented on SOLR-11078:
---------------------------------
We have 3 identical production servers. Two are running Solr 6.4.2 where the
schema is still using Trie* fields. The 3rd server where we have the
performance degradation running any version of Solr since 6.4.2 has now been
upgraded to Solr 7.1.0 where Trie*-fields have been changed to *Point-fields.
Index has been rebuild and Solr performance is worse compared to Solr 6.4.2.
All servers are running:
- 10 GB RAM, 4 CPUs
- CentOS 7.3.1611
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Solr Heap -Xms/Xmx is set to 4GB. GC tuning as per attached solr.in.sh (the
settings are the same between 6.4.2 and 7.1.0)
- Two Solr indices: tradesearch (5.5m documents and 3,4GB size) and
searchsuggestions (2.7m documents and 700MB size)
Switching Solr 7.1.0 into production load:
* Load average (via top) shot to 19 and after a few minutes "settled" at 12 (on
6.4.2 it is about 2.2 - 4)
* Average query time is about 230ms - this is 4 times slower compared to 6.4.2
(on 6.4.2 it is about 50ms)
* A specific query (we know it is generally slow) runs between 8-10 seconds (on
6.4.2 it would that about 1.3 seconds)
I can switch 7.1.0 into parallel production tests to collate more data. Any
help is appreciated.
> Solr query performance degradation since Solr 6.4.2
> ---------------------------------------------------
>
> Key: SOLR-11078
> URL: https://issues.apache.org/jira/browse/SOLR-11078
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 6.6
> Environment: * CentOS 7.3 (Linux zasolrm03 3.10.0-514.26.2.el7.x86_64
> #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
> * Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> * 4 CPU, 10GB RAM
> Running Solr 6.6.0 with the following JVM settings:
> java -server -Xms4G -Xmx4G -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/home/prodza/solrserver/../logs/solr_gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
> -Dsolr.log.dir=/home/prodza/solrserver/../logs -Djetty.port=8983
> -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=SAST
> -Djetty.home=/home/prodza/solrserver/server
> -Dsolr.solr.home=/home/prodza/solrserver/../solr
> -Dsolr.install.dir=/home/prodza/solrserver
> -Dlog4j.configuration=file:/home/prodza/solrserver/../config/log4j.properties
> -Xss256k -Xss256k -Dsolr.log.muteconsole
> -XX:OnOutOfMemoryError=/home/prodza/solrserver/bin/oom_solr.sh 8983
> /home/prodza/solrserver/../logs -jar start.jar --module=http
> Reporter: bidorbuy
> Priority: Major
> Attachments: compare-6.4.2-6.6.0.png, core-admin-tradesearch.png,
> jvm-stats.png, schema.xml, screenshot-1.png, screenshot-2.png,
> solr-sample-warning-log.txt, solrconfig.xml
>
>
> We are currently running 2 separate Solr servers - refer to screenshots:
> * zasolrm02 is running on Solr 6.4.2
> * zasolrm03 is running on Solr 6.6.0
> Both servers have the same OS / JVM configuration and are using their own
> indexes. We round-robin load-balance through our Tomcats and notice that
> Since Solr 6.4.2 performance has dropped. We have two indices per server
> "searchsuggestions" and "tradesearch". There is a noticeable drop in
> performance since Solr 6.4.2.
> I am not sure if this is perhaps related to metric collation or other
> underlying changes. I am not sure if other high transaction users have
> noticed similar issues.
> *1) zasolrm03 (6.6.0) is almost twice as slow on the tradesearch index:*
> !compare-6.4.2-6.6.0.png!
> *2) This is also visible in the searchsuggestion index:*
> !screenshot-1.png!
> *3) The Tradesearch index shows the biggest difference:*
> !screenshot-2.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
