[
https://issues.apache.org/jira/browse/SOLR-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240871#comment-16240871
]
David Smiley commented on SOLR-11078:
-------------------------------------
bq. Is there more documentation available about *Point-types?
I'm not sure. I'll update the ref guide after you take the changes for a whirl
to confirm what I expect will happen.
bq. I am unsure what it means from a performance perspective that Point-types
do not support FieldCache and as such we need to use doc-values for faceting to
be supported.
Curious; can you provide a reference to where you've read that about the
FieldCache? It's right but just wanted to see where that advise is; I didn't
notice it in the ref guide. Any way... I suspect you've _already_ (in 6.x)
been using docValues=true on appropriate fields since you didn't mention to the
contrary. So there's no change for you in this respect. Basically, know that
Solr needs fast access to values on a doc/field basis for certain non-search
features. Generally, the ideal way to make that available is to set
docValues=true on the field. Alternatively if you don't, and if the field type
supports "uninversion" (Trie fields & StrField do, Point fields don't) then
Solr will use it's so-called FieldCache which un-inverts the index=true data
into memory. Lets say you weren't using docValues at all and now you do. The
net perceived change is
* smaller java heap requirements; this data is accessed off-heap in memory
mapped IO instead. So you still need same RAM roughly but Java GC can have an
easier time.
* better realtime search. The warmup queries (generally user-configured to run
after commit in solrconfig) should be faster to complete if they use faceting,
sorting etc. since they won't need to trigger the uninversion of index
structures into memory. If you don't have warmup, then the first user's search
query to come along after a commit used to be hit with the uninversion penalty
and now it won't (so faster queries at this 99 percentile).
* larger disk requirements to hold the docValues data
General access performance isn't necessarily better though; in some cases it
has been shown to be worse :-/
bq. One idea we had is to change all non-range fields to StrField (one by one)
and measure performance impact and then start switching *Point-fields. I am not
sure if there is a better/scientific way to go about this (analysing the query
performance is difficult as queries are typically some text-search with
filtering/faceting - so not many unique queries where it is easy to show some
trends).
Benchmarking/analysis can be a real time sink; I suppose what you should do
depends on what you hope to get out of it. Assuming you hope to not use
deprecated Trie fields... then yes moving away from them is one goal. And
getting equal or better performance is another goal. I doubt you need to get
any more precise than that unless you are digging deeper to discover why one of
those goals isn't met (gets slower). As you've indicated switching the
"non-range" usages to StrField is one step. My expectation of the result is
that it'll be a non-event. StrField loses the typed ergonomics of Trie but
otherwise should be very similar for non-range use. I wouldn't bother changing
one field at a time and measuring in-between... that seems like a lot of time;
although if you discover some unexpected bad results then you could do this to
help deduce which field & query exhibits the problem. RE the range fields,
yep, them from TrieFields to PointFields (all at once) is the next step. This
one may be interesting; hopefully nothing gets worse. If it's better than
declare success and move on!
> Solr query performance degradation since Solr 6.4.2
> ---------------------------------------------------
>
> Key: SOLR-11078
> URL: https://issues.apache.org/jira/browse/SOLR-11078
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search, Server
> Affects Versions: 6.6, 7.1
> Environment: * CentOS 7.3 (Linux zasolrm03 3.10.0-514.26.2.el7.x86_64
> #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
> * Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> * 4 CPU, 10GB RAM
> Running Solr 6.6.0 with the following JVM settings:
> java -server -Xms4G -Xmx4G -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/home/prodza/solrserver/../logs/solr_gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
> -Dsolr.log.dir=/home/prodza/solrserver/../logs -Djetty.port=8983
> -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=SAST
> -Djetty.home=/home/prodza/solrserver/server
> -Dsolr.solr.home=/home/prodza/solrserver/../solr
> -Dsolr.install.dir=/home/prodza/solrserver
> -Dlog4j.configuration=file:/home/prodza/solrserver/../config/log4j.properties
> -Xss256k -Xss256k -Dsolr.log.muteconsole
> -XX:OnOutOfMemoryError=/home/prodza/solrserver/bin/oom_solr.sh 8983
> /home/prodza/solrserver/../logs -jar start.jar --module=http
> Reporter: bidorbuy
> Attachments: compare-6.4.2-6.6.0.png, core-admin-tradesearch.png,
> jvm-stats.png, schema.xml, screenshot-1.png, screenshot-2.png,
> solr-6-4-2-schema.xml, solr-6-4-2-solrconfig.xml, solr-7-1-0-managed-schema,
> solr-7-1-0-solrconfig.xml, solr-71-vs-64.png, solr-sample-warning-log.txt,
> solr.in.sh, solrconfig.xml
>
>
> We are currently running 2 separate Solr servers - refer to screenshots:
> * zasolrm02 is running on Solr 6.4.2
> * zasolrm03 is running on Solr 6.6.0
> Both servers have the same OS / JVM configuration and are using their own
> indexes. We round-robin load-balance through our Tomcats and notice that
> Since Solr 6.4.2 performance has dropped. We have two indices per server
> "searchsuggestions" and "tradesearch". There is a noticeable drop in
> performance since Solr 6.4.2.
> I am not sure if this is perhaps related to metric collation or other
> underlying changes. I am not sure if other high transaction users have
> noticed similar issues.
> *1) zasolrm03 (6.6.0) is almost twice as slow on the tradesearch index:*
> !compare-6.4.2-6.6.0.png!
> *2) This is also visible in the searchsuggestion index:*
> !screenshot-1.png!
> *3) The Tradesearch index shows the biggest difference:*
> !screenshot-2.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
