Re: Need urgent help -- High cpu on solr

[Zisis T.]

The values you have for the caches and the maxwarmingsearchers do not look
like the default. Cache sizes are 512 for the most part and
maxwarmingsearchers are 2 (if not limit them to 2)

Sudden CPU spikes probably indicate GC issues. The #  of documents you have
is small, are they huge documents? The # of collections is OK in general but
since they are crammed in 5 Solr nodes the memory requirements might be
bigger. Especially if filter and the other caches get populated with 50K
entries. 

I'd first go through the GC activity to make sure that this is not causing
the issue. The fact that you lose some Solr servers is also an indicator of
large GC pauses that might create a problem when Solr communicates with
Zookeeper. 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Analytics for Solr logs

[Zisis T.]

Thanks Alexandre, silly me. I though 8.4.1 was recent enough...



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Analytics for Solr logs

[Zisis T.]

I've stumbled upon
https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc
which looks very interesting for getting insights into the Solr logs.

I cannot find though postlogs command inside the Solr bin dir (there is post
command though) nor a way to create the logs collection. I've looked into
solr-8.4.1 and solr-7.5.0 but could not find anything.

1) Is this still supported?
2) Where can I find the logs collection configuration? How can I create it?
3) Is post the same command as postlogs?

Thanks 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr heap Old generation grows and it is not recovered by G1GC

[Zisis T.]

Hi Reinaldo, 

Glad that helped. I've had several sleepless nights with Solr clusters
failing spectacularly in production due to that but I still cannot say that
the problem is completely away.

Did you check in the heap dump if you have cache memory leaks as described
in https://issues.apache.org/jira/browse/SOLR-12743? 

Say you have 4 cache instances (filterCache, documentCache etc) per core and
you have 5 Solr cores you should not see more than 20 CaffeineCache
instances in your dump. 

Unfortunately I still cannot determine what exactly triggers this memory
leak although since I removed the maxRAMMB setting I've not seen similar
behavior for more than a month now in production. 

The weird thing is that I was running on Solr 7.5.0 for quite some time
without any issues and it was at some point in time that those problems
started appearing...



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr heap Old generation grows and it is not recovered by G1GC

[Zisis T.]

I have faced similar issues and the culprit was filterCache when using
maxRAMMB. More specifically on a sharded Solr cluster with lots of faceting
during search (which makes use of the filterCache in a distributed setting)
I noticed that maxRAMMB value was not respected. I had a value of 300MB set
but I witnessed an instance sized a couple of GBs in a heap dump at some
point. The thing that I found was that because the keys of the Map
(BooleanQuery or something if I recall correctly) was not implementing the
Accountable interface it was NOT taken into account when calculating the
cache's size. But all that was on a 7.5 cluster using FastLRUCache. 

There's also https://issues.apache.org/jira/browse/SOLR-12743 on caches
memory leak which does not seem to have been fixed yet although the trigger
points of this memory leak are not clear. I've witnessed this as well on a
7.5 cluster with multiple (>10) filter cache objects for a single core each
holding from a few MBs to GBs. 

Try to get a heap dump from your cluster, the truth is almost always hidden
there. 

One workaround which seems to alleviate the problem is to check you running
Solr cluster and see in reality how many cache entries actually give you a
good hit ratio and get rid of the maxRAMMB attribute. Play only with the
size. 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Java version 11 for solr 7.5?

[Zisis T.]

Jeff Courtade wrote
> Can we use GC1 garbage collection yet or do we still need to use CMS?

I believe you should be safe to go with G1. We've applied it in in a Solr
6.6 cluster with 10 shards, 3 replicas per shard and an index of about 500GB
(1,5T counting all replicas) and it works extremely well (throughput > 99%).
The use-case includes complex search queries and faceting. 
There is also this post you can use as a starting point
http://blog.cloudera.com/blog/2017/06/apache-solr-memory-tuning-for-production/




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Timeout waiting for connection from pool

[Zisis T.]

If you are having an issue when increasing the search request rate you should
have a look at maxConnectionsPerHost, I believe maxUpdateConnectionsPerHost
is related to indexing. 
You can modify your solr.xml as follows (I believe it's not clear from the
official documentation, I had to go through the code to find out about
maxConnectionsPerHost )


15000
12
500
  

Another thing I'd check is if you have GC pauses, it might affect your
performance.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Caching of dynamic external file fields

[Zisis T.]

In Solr there's /ExternalFileFieldReloader/ which is responsible for caching
the contents of external files whenever a new searcher is being warmed up. 

It happens that I've defined a dynamic field to be used as an
/ExternalField/ as in 
/* */ 

If you have a look inside the code /ExternalFileFieldReloader/ there's a
loop over the explicit schema fields through /schema.getFields()/ which
means that dynamic external fields do not get cached. 

Of course I did notice that once those files started getting bigger (~50MB)
and searches started timing out. 

Q1 : Is there a specific reason why only explicit fields are cached?
Q2 : I have extended /ExternalFileFieldReloader/ and added the capability to
also cache dynamic external file field contents and it seems to be working
fine. Does anyone think this is useful to make it into Solr codebase?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Facets OutOfMemoryException

[Zisis T.]

I believe that things like the following will affect faceting memory
requirements
-> how many fields do you facet on 
-> what is the cardinality of each one of them 
-> What is you QPS rate

but 2GB for 27M documents seems too low. Did you try to increase the memory
on Solr's JVM?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr nodes going into recovery mode and eventually failing

[Zisis T.]

shamik wrote
> I was not aware of maxRamMB parameter, looks like it's only available for
> queryResultCache. Is that what you are referring to? Can you please share
> your cache configuration?

I've setup filterCache entry inside solrconfig.xml as follows

**

I had a look inside FastLRUCache code and saw that maxRamMB has precedence
over the size. I can also confirm that I had more than 512 entries inside
the cache, so the above will work. 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr nodes going into recovery mode and eventually failing

[Zisis T.]

I'll post my experience too, I believe it might be related to the low
FilterCache hit ratio issue. Please let me know if you think I'm off topic
here to create a separate thread.

I've run search stress tests on a 2 different Solr 6.5.1 installations
sending Distributed search queries with facets (using facet.field, faceted
fields have docValues="true")

*1-shard*
1 shard of ~200GB/8M docs
FilterCache with default 512 entries. 90% hit ratio

*2-shards*
2 shards of ~100GB/4M docs each
FilterCache with default 512 entries. 10% hit ratio. Huge drop in search
performance

Noticed that the majority of the FilterCache entries look like "filters on
facet terms" and instead of a FixedBitSet which size is equal to the # of
docs in the shard, it contains
an int[] of the matching docids

  Key   Value
-
FacetField1:Term1   ->  int[] of matching docids
FacetField1:Term2   ->  int[] of matching docids
FacetField2:Term3   ->  int[] of matching docids
FacetField2:Term4   ->  int[] of matching docids

Given that Field1 and Field2 are high cardinality fields there are too many
keys in the cache but with few matched documents in most of the cases.
Therefore since the cache values do not need so much memory, I ended up
using *maxRamMB*=120 which in my case gives ~80% hit ratio allowing more
entries in the cache and better control over consumed memory. 

This has been previously discussed here too
http://lucene.472066.n3.nabble.com/Filter-cache-pollution-during-sharded-edismax-queries-td4074867.html#a4162151

Is this "overuse" of FilterCache normal in distributed search? 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Measuring time spent in analysis and writing to index

[Zisis T.]

Another thing you can do - and which has helped me in the past quite a few
times - is to just run JVisualVM, attach to Solr's Java process and enable
the CPU sampler under the Sampler tab.

As you run indexing the methods that most time is spent on will appear near
the top. 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Measuring time spent in analysis and writing to index

[Zisis T.]

I've worked in the past for a Solr 5.x custom plugin using AspectJ to track
the # of calls as well as the time spent inside /incrementToken()/ of all
Tokenizers and Filters used during indexing. I could get stats per Solr
indexing thread, not per indexing request though. In any case you could spot
the filter/tokenizer where most of the indexing time was spent. 
Not sure if there's something similar in Solr 6.x or 7.x.

You can see a sample of the output here
 




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Multi tenant setup

[Zisis T.]

We are talking about fewer collections,so that won't be an issue. 

The problem comes when - using the proposed setup - I want to send a query
across all those collections and get properly ranked results. Each
collection has its own IDF etc, so the scores are not comparable. This means
that most probably results from one collection will dominate the results. 

This led me to try the /DistributedIDF/ configuration but this did not work
either due to the issues described in the link of the original post. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Multi-tenant-setup-tp4340377p4340421.html
Sent from the Solr - User mailing list archive at Nabble.com.

Multi tenant setup

[Zisis T.]

I'm trying to setup a multi-tenant Solr cluster (v6.5.1) which must meet the
following requirements. The tenants are different customers with similar
type of data.

* Ability to query per client but also across all clients
* Don't want to hit all shards for all type of requests (per client, across
clients)
* Don't want to have everything under a single multi-sharded collection to
avoid a SPOF and maintenance headaches 
   (e.g. a schema change will force an all-client reindexing. single huge
backup/restore)
* Ability to semi-support different schemas.

Based on the above I ruled out the following setups 
* Single multi-sharded collection for all clients and all its variations
(e.g. multiple clients in a singe shard)
* One collection per client 

My preference lies in a setup like the following
* Create a limited # of collections
* Split the clients in the collections created above based on some criteria
(size, content-type)
* Client specific requests will be limited in a single collection
* Across clients requests will target a limited # of collections (using
=col_1,col_2,col_3)

The approach above meets the requirements posted above but the issue that is
blocking me is the Distributed IDF not working properly across collections.
(Check comment#3, bullet#2 of
http://lucene.472066.n3.nabble.com/Distributed-IDF-in-inter-collections-distributed-queries-td4317519.html)


-> Do you see anything wrong with my assumptions/approach above? Are there
any alternatives besides having separate clusters for the search across
clients and the individual clients?
-> Is it safe to go with a single collection? If it is, I still need to
handle the possible different schemas per client somehow.
-> Is there a way to enforce local stats when quering a single collection
and use global stats only when querying across collections? (see link above)

Thanks



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Multi-tenant-setup-tp4340377.html
Sent from the Solr - User mailing list archive at Nabble.com.