Re: Default Index config

[mganeshs]

To add it further, in 6.5.1, while indexing... even sometimes one of solr
node goes down for a while and comes up automatically. During those period
all our calls to index fails. Even in the Solr admin UI, we can see node not
being active for a while and coming up again.

All these happens in 4 core machine ( r4-xlarge). If we move to r4-2xlarge (
8 core machine )... everything goes smooth without any issue. Even CPU also
stays with 50%.

Is that mean that we need 8 core machine to run the index and query with
such a data and heavy indexing rate ? 





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Default Index config

[mganeshs]

Hi Shawn, 

We found following  link

  
where its mentioned like in 6.2.1 it's faster where as in 6.6 its slower.
Keep this, we too tried with 6.2.1 in our performance environment and we
found that CPU usage came down to 60 to 70% where as in 6.5.1 it was always
more than 95% 

Settings are same and data size and indexing speed remains same. Pls check
the  JVM snapshot

  
when we index using 6.2.1


Following is the  snapshot

 
taken with 6.5.1

Is there any reason why such a huge difference with CPU usage patterns
between 6.2.1 and 6.5.1 ? 

Can we do something in 6.5.1 to make it as 6.2.1? Because we don't want to
downgrade to 6.2.1 from 6.5.1. 

Let us know your thoughts on this.

Thanks and Regards,





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Default Index config

[mganeshs]

Hi Shawn,

Thanks for the reply. 

Yes we use only one solr client. Though collection name is passed in the
function, we are using same client for now.

Regarding merge config, after reading lot of forums and listening to
presentation of revolution 2017, idea is to reduce the merge frequency, so
that CPU usage pattern will come down from 100 to 70% for a while and only
when merges happens it will go to 100% ( where as now it's always above 95%
) which we see it as not a good sign of CPU always more than 95% since we
run other components as well in this server. So to reduce the merge
frequency,  i was trying that. 

Thanks for sharing your config, will try to check with that too and post you
an update on the result. 

Thanks and Regards,




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Default Index config

[Shawn Heisey]

On 4/9/2018 4:04 AM, mganeshs wrote:

Regarding CPU high, when we are troubleshooting, we found that Merge threads
are keep on running and it's take most CPU time ( as per Visual JVM ).


With a one second autoSoftCommit, nearly constant indexing will produce 
a lot of very small index segments.  Those index segments will have to 
be merged eventually.  You have increased the merge policy numbers which 
will reduce the total number of merges, but each merge is going to be 
larger than it would with defaults, so it's going to take a little bit 
longer.  This isn't too big a deal with first-level merges, but at the 
higher levels, they do get large -- no matter what the configuration is.



*Note*: following is the code snippet we use for indexing / adding solr
document in batch per collection

/for (SolrCollectionList solrCollection : SolrCollectionList.values()) {
CollectionBucket collectionBucket = getCollectionBucket(solrCollection);
List solrInputDocuments =
collectionBucket.getSolrInputDocumentList();
String collectionName = collectionBucket.getCollectionName();
try {
if(solrInputDocuments.size() > 0) {
CloudSolrClient solrClient =
PlatformIndexManager.getInstance().getCloudSolrClient(collectionName);
solrClient.add(collectionName, solrInputDocuments);
}
}/

*where solrClient is created as below
*
/this.cloudSolrClient = new
CloudSolrClient.Builder().withZkHost(zooKeeperHost).withHttpClient(HttpClientUtil.HttpClientFactory.createHttpClient()).build();
this.cloudSolrClient.setZkClientTimeout(3);
/


Is that code running on the Solr server, or on a different machine?  Are 
you creating a SolrClient each time you use it, or have you created 
client objects that get re-used?


You don't need a different SolrClient object for each collection.  Your 
"getCloudSolrClient" method takes a collection name, which suggests 
there might be a different client object for each one.  Most of the 
time, you need precisely one client object for the entire application.



Hard commit is kept as automatic and set to 15000 ms.
In this process, we also see, when merge is happening, and already
maxMergeCount ( default one ) is reached, commits are getting delayed and
solrj client ( where we add document ) is getting blocked and once once of
Merge thread process the merge, then solrj client returns the result.
How do we avoid this blocking of solrj client ? Do I need to go out of
default config for this scenario? I mean change the merge factor
configuration ?

Can you suggest what would be merge config for such a scenario ? Based on
forums, I tried to change the merge settings to the following,


What are you trying to accomplish by changing the merge policy?  It's 
fine to find information for a config on the Internet, but you need to 
know what that config *does* before you use it, and make sure it aligns 
with your goals.  On mine, I change maxMergeAtOnce and segmentsPerTier 
to 35, and maxMergeAtOnceExplicit to 105.  I know exactly what I'm 
trying to do with this config -- reduce the frequency of merges.  Each 
merge is going to be larger with this config, but they will happen less 
frequently.  These three settings are the only ones that I change in my 
merge policy.  Changing all of the other settings that you have changed 
should not be necessary.  I make one other adjustment in this area -- to 
the merge scheduler.



In same solr node, we have multiple index / collection. In that case,
whether TieredMergePolicyFactory will be right option or for multiple
collection in same node we should go for other merge policy ( like LogByte
etc )


TieredMergePolicy was made the default policy after a great deal of 
testing and discussion by Lucene developers.  They found that it works 
better than the others for the vast majority of users.  It is likely the 
best choice for you too.


These are the settings that I use in indexConfig to reduce the impact of 
merges on my indexing:


  
    35
    35
    105
  
  
    1
    6
  

Note that this config is designed for 6.x and earlier.  I do not know if 
it will work in 7.x.  It probably needs to be adjusted to the new 
Factory config.  You can use it as a guide, though.


Thanks,
Shawn

Re: Default Index config

[mganeshs]

Hi Shawn,

Regarding CPU high, when we are troubleshooting, we found that Merge threads
are keep on running and it's take most CPU time ( as per Visual JVM ). GC is
not causing any issue as we use the default GC and also tried with G1 as you
suggested over  here
  

Though it's only background process, we are suspecting whether it's causing
CPU to go high. 

Since we are using SOLR as real time indexing of data and depending on its
result immd. to show it in UI as well. So we keep adding document around 100
to 200 documents in parallel in a sec. Also it would be in batch of 20 solr
documents list in one add... 

*Note*: following is the code snippet we use for indexing / adding solr
document in batch per collection

/for (SolrCollectionList solrCollection : SolrCollectionList.values()) {
CollectionBucket collectionBucket = getCollectionBucket(solrCollection);
List solrInputDocuments =
collectionBucket.getSolrInputDocumentList();
String collectionName = collectionBucket.getCollectionName();
try {
if(solrInputDocuments.size() > 0) {
CloudSolrClient solrClient =
PlatformIndexManager.getInstance().getCloudSolrClient(collectionName);
solrClient.add(collectionName, solrInputDocuments);
}
}/

*where solrClient is created as below
*
/this.cloudSolrClient = new
CloudSolrClient.Builder().withZkHost(zooKeeperHost).withHttpClient(HttpClientUtil.HttpClientFactory.createHttpClient()).build();
this.cloudSolrClient.setZkClientTimeout(3);
/

Hard commit is kept as automatic and set to 15000 ms.
In this process, we also see, when merge is happening, and already
maxMergeCount ( default one ) is reached, commits are getting delayed and
solrj client ( where we add document ) is getting blocked and once once of 
Merge thread process the merge, then solrj client returns the result.
How do we avoid this blocking of solrj client ? Do I need to go out of
default config for this scenario? I mean change the merge factor
configuration ? 

Can you suggest what would be merge config for such a scenario ? Based on
forums, I tried to change the merge settings to the following,


30
30
30
2048
512
0.1
2048
2.0
10.0


But couldn't see any much change in the behaviour.

In same solr node, we have multiple index / collection. In that case,
whether TieredMergePolicyFactory will be right option or for multiple
collection in same node we should go for other merge policy ( like LogByte
etc ) 


Can you throw some light on this aspects ?
Regards,

 Regarding auto commit, we discussed lot with our product owners and atlast
> we are forced to keep it to 1sec and we couldn't increase further. As this
> itself, sometimes our customers says that they have to refresh their pages
> for couple of times to get the update from solr. So we can't increase
> further.

I understand pressure from nontechnical departments for very low 
response times. Executives, sales, and marketing are usually the ones 
making those kinds of demands. I think you should push back on that 
particular requirement on technical grounds.

A soft commit interval that low *can* contribute to performance issues.  
It doesn't always cause them, I'm just saying that it *can*.  Maybe 
increasing it to five or ten seconds could help performance, or maybe it 
will make no real difference at all.

> Yes. As of now only solr is running in that machine. But intially we were
> running along with hbase region servers and was working fine. But due to
> CPU
> spikes and OS disk cache, we are forced to move solr to separate machine.
> But just I checked, our solr data folder size is coming only to 17GB. 2
> collection has around 5GB and other are have 2 to 3 GB of size. If you say
> that only 2/3 of total size comes to OS disk cache, in top command VIRT
> property it's always 28G, which means more than what we have. Why is
> that...
> Pls check that top command & GC we used in this  doc
> ;

The VIRT memory should be about equivalent to the RES size plus the size 
of all the index data on the system.  So that looks about right.  The 
actual amount of memory allocated by Java for the heap and other memory 
structures is approximately equal to RES minus SHR.

I am not sure whether the SHR size gets counted in VIRT. It probably 
does.  On some Linux systems, SHR grows to a very high number, but when 
that happens, it typically doesn't reflect actual memory usage.  I do 
not know why this sometimes happens.That is a question for Oracle, since 
they are the current owners of Java.

Only 5GB is in the buff/cache area.  The system has 13GB of free 
memory.  That system is NOT low on memory.

With 4 CPUs, a load average in the 3-4 range is an indication that the 
ser

Re: Default Index config

[Shawn Heisey]

On 3/28/2018 9:44 PM, mganeshs wrote:

Regarding auto commit, we discussed lot with our product owners and atlast
we are forced to keep it to 1sec and we couldn't increase further. As this
itself, sometimes our customers says that they have to refresh their pages
for couple of times to get the update from solr. So we can't increase
further.


I understand pressure from nontechnical departments for very low 
response times. Executives, sales, and marketing are usually the ones 
making those kinds of demands. I think you should push back on that 
particular requirement on technical grounds.


A soft commit interval that low *can* contribute to performance issues.  
It doesn't always cause them, I'm just saying that it *can*.  Maybe 
increasing it to five or ten seconds could help performance, or maybe it 
will make no real difference at all.



Yes. As of now only solr is running in that machine. But intially we were
running along with hbase region servers and was working fine. But due to CPU
spikes and OS disk cache, we are forced to move solr to separate machine.
But just I checked, our solr data folder size is coming only to 17GB. 2
collection has around 5GB and other are have 2 to 3 GB of size. If you say
that only 2/3 of total size comes to OS disk cache, in top command VIRT
property it's always 28G, which means more than what we have. Why is that...
Pls check that top command & GC we used in this  doc



The VIRT memory should be about equivalent to the RES size plus the size 
of all the index data on the system.  So that looks about right.  The 
actual amount of memory allocated by Java for the heap and other memory 
structures is approximately equal to RES minus SHR.


I am not sure whether the SHR size gets counted in VIRT. It probably 
does.  On some Linux systems, SHR grows to a very high number, but when 
that happens, it typically doesn't reflect actual memory usage.  I do 
not know why this sometimes happens.That is a question for Oracle, since 
they are the current owners of Java.


Only 5GB is in the buff/cache area.  The system has 13GB of free 
memory.  That system is NOT low on memory.


With 4 CPUs, a load average in the 3-4 range is an indication that the 
server is busy.  I can't say for sure whether it means the server is 
overloaded.  Sometimes the load average on a system that's working well 
can go higher than the CPU count, sometimes a load average well below 
the CPU count is shown on a system with major performance issues.  It's 
difficult to say.  The instantaneous CPU usage on the Solr process in 
that screenshot is 384 percent.  Which means that it is exercising the 
CPUs hard. But this might be perfectly OK.  96.3 percent of the CPU is 
being used by user processes, a VERY small amount is being used by 
system, and the iowait percentage is zero.  Typically servers that are 
struggling will have a higher percentage in system and/or iowait, and I 
don't see that here.



Queries are quiet fast, most of time simple queries with fq. Regarding
index, during peak hours, we index around 100 documents in a second in a
average.


That's good.  And not surprising, given how little memory pressure and 
how much free memory there is.  An indexing rate of 100 per second 
doesn't seem like a lot of indexing to me, but for some indexes, it 
might be very heavy.  If your general performance is good, I wouldn't be 
too concerned about it.



Regarding release, initially we tried with 6.4.1 and since many discussions
over here, mentioned like moving to 6.5.x will solve lot of performance
issues etc, so we moved to 6.5.1. We will move to 6.6.3 in near future.


The 6.4.1 version had a really bad bug in it that killed performance for 
most users.  Some might not have even noticed a problem, though.  It's 
difficult to say for sure whether it would be something you would 
notice, or whether you would see an increase in performance by upgrading.



Hope I have given enough information. One strange thing is that, CPU and
memory spike are not seen when we move to r4.xlarge to r4.2xlarge ( which is
8 core with 60 GB RAM ). But this would not be cost effective. What's making
CPU and memory to go high in this new version ( due to doc values )? If I
switch off docvalues will CPU & Memory spikes will get reduced ?


Overall memory usage (outside of the Java heap) looks great to me.  CPU 
usage is high, but I can't tell if it's TOO high. As a proof of concept, 
I think you should try raising autoSoftCommit to five seconds.  If 
maxDocs is configured on either autoCommit or autoSoftCommit, remove it 
so that only maxTime is there, regardless of whether you actually change 
maxTime.  If raising autoSoftCommit makes no real difference, then the 1 
second autoSoftCommit probably isn't a worry.  I bet if you raised it to 
five seconds, most users would never notice anything different.


If you want to provide a G

Re: Default Index config

[mganeshs]

Hi Shawn,

Thanks again for detailed reply.

Regarding auto commit, we discussed lot with our product owners and atlast
we are forced to keep it to 1sec and we couldn't increase further. As this
itself, sometimes our customers says that they have to refresh their pages
for couple of times to get the update from solr. So we can't increase
further.

We have kept openSearcher, it's as typical configuration & recommended only.

Yes. As of now only solr is running in that machine. But intially we were
running along with hbase region servers and was working fine. But due to CPU
spikes and OS disk cache, we are forced to move solr to separate machine. 
But just I checked, our solr data folder size is coming only to 17GB. 2
collection has around 5GB and other are have 2 to 3 GB of size. If you say
that only 2/3 of total size comes to OS disk cache, in top command VIRT
property it's always 28G, which means more than what we have. Why is that...
Pls check that top command & GC we used in this  doc

  

Yes we are using solr cloud only.

Queries are quiet fast, most of time simple queries with fq. Regarding
index, during peak hours, we index around 100 documents in a second in a
average. 

I also shared the CPU utilization in the same  doc

  

Regarding release, initially we tried with 6.4.1 and since many discussions
over here, mentioned like moving to 6.5.x will solve lot of performance
issues etc, so we moved to 6.5.1. We will move to 6.6.3 in near future. 

Hope I have given enough information. One strange thing is that, CPU and
memory spike are not seen when we move to r4.xlarge to r4.2xlarge ( which is
8 core with 60 GB RAM ). But this would not be cost effective. What's making
CPU and memory to go high in this new version ( due to doc values )? If I
switch off docvalues will CPU & Memory spikes will get reduced ? 

Let me know...

 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Default Index config

[Shawn Heisey]

On 3/27/2018 9:35 PM, mganeshs wrote:

I am using the default configuration, as all solr experts say default one
suits for most of the cases and so following the defaults. We changed only
the commits part and using 15000 for hard commits and 1 sec for soft commit.
All other setting like locking, deleting policy, merge, directory, etc are
left to default ones.


One second for autoSoftCommit has a tendency to cause a lot of issues.  
That interval should be set as large as you can tolerate for change 
visibility.  I personally would want to see a setting there of at least 
one minute, but I know that this is not fast enough for a lot of users.


Does autoCommit have openSearcher set to false?  This is a typical 
configuration, and recommended.



One strange thing we noticed after moving from solr 5.x to solr 6.5.1 is
that CPU and RAM usage is increased drastically. We have two solr nodes, one
for data and another for replica. It's EC2 r4.xlarge machine. We have almost
6 collection  and each carries around 5GB of data in average and in couple
of collection we have frequent updates too.


That kind of instance has 30.5 GB of total memory.  With 8GB assigned to 
the heap, that leaves about 22GB for everything else.  If Solr is the 
only thing running on the machine, and your numbers mean that each 
server has about 30GB of index data, then that means you can get about 
two thirds of the index into the OS disk cache.   Usually this is enough 
for decent performance, but that's not the case for everyone.  If this 
is an accurate picture of your Solr install, I am not really worried 
about total memory size.  I can't make a guess about it unless I have 
accurate information.


The way you're phrasing things, it sounds like you're running in 
SolrCloud mode.  Is that correct?


Since you're probably running Solr on Linux, can you get me a screenshot 
from the "top" program?  It must be specifically that program -- don't 
use an alternate utility like "htop".


Run top, press shift-M to sort the display by memory, and then grab a 
screenshot or photo of the display.  Share that file somewhere and 
provide a URL to access it.  With that info, we can get a good picture 
of system health.  Be sure that when that "top" display is grabbed, that 
the system is experiencing whatever problems you're trying to fix.


What do you know about your query load and how fast the data is being 
indexed?



In solr 5.x we didn't feel this much of RAM and CPU usages. CPU is always 80
to 90% even if we are trying to index or update some 50 docs at one shot and
RAM it occupies whatever we give. We started with 8GB of Heap. But its
always 8GB.  Initially we were using CMS GC and tried with G1 GC. Only
difference is that, In case of CMS, even after starting solr, cpu goes to
80%, where as in G1, after started solr it's around 10% and when load comes
( around 100 to 200 docs  in 5 mins ) it's goes 90% ( in both CMS and G1 )


The default garbage collection tuning does a pretty good job most of the 
time.  For some people, G1 works even better, but this can only happen 
if it is tuned.  Simply turning on G1GC and leaving it untuned will 
probably get you WORSE performance than leaving the GC tuning at the 
settings that the project has provided.



When we tried to profile the solr process, we found that merging is keep on
happening.


Can you elaborate more on exactly what you observed, and what 
conclusions you came to based on that information?



This spikes of CPU and memory, we are seeing only after moving to 6.5.1. Is
that stable version ? moving to latest stable will solve this issue or we
miss something wrt configurations ? Do we need to change the solr default
config ?


Virtually all releases are considered stable at the time of release.  If 
they weren't, then they wouldn't be released!  I'm only aware of two 
times that versions were released without being called "stable."  Those 
were the 4.0-ALPHA and 4.0-BETA releases.  There have been no ALPHA or 
BETA releases since then.


Pretty much every version is later revealed to have bugs, and sometimes 
those problems are big enough that the release could be called 
unstable.  But that is a determination made AFTER release, not before.


6.5.1 was announced on 27 April 2017.  It is a bugfix release, mostly 
identical to 6.5.0, which was announced exactly one month earlier, on 27 
March 2017.  Which makes it one year old.


There are a number of bugfixes in 6.5.1 compared to 6.5.0, but the two 
are substantially similar.


http://mail-archives.apache.org/mod_mbox/www-announce/201704.mbox/%3CCAKUpjcQiYLZ1a+ZM=ohuxjywlj9vnub8q9b_ilrvmkididv...@mail.gmail.com%3E

If you're going to run 6.x, then you should run 6.6.3.  The latest 
version currently out is 7.2.1.  Version 7.3.0 is just around the corner 
-- release is underway right now.  I haven't looked through the 
changelogs, so I don't really know what's coming.


Thanks,
Shawn

Re: Default Index config

[mganeshs]

Hi Shawn,

Thanks for detail mail. Yes I am behind the IndexConfig only.

Regarding 5GB size of collection, it's not one document. It has almost 3M of
docs in that collection. 

I am using the default configuration, as all solr experts say default one
suits for most of the cases and so following the defaults. We changed only
the commits part and using 15000 for hard commits and 1 sec for soft commit. 
All other setting like locking, deleting policy, merge, directory, etc are
left to default ones.

One strange thing we noticed after moving from solr 5.x to solr 6.5.1 is
that CPU and RAM usage is increased drastically. We have two solr nodes, one
for data and another for replica. It's EC2 r4.xlarge machine. We have almost
6 collection  and each carries around 5GB of data in average and in couple
of collection we have frequent updates too. 

In solr 5.x we didn't feel this much of RAM and CPU usages. CPU is always 80
to 90% even if we are trying to index or update some 50 docs at one shot and
RAM it occupies whatever we give. We started with 8GB of Heap. But its
always 8GB.  Initially we were using CMS GC and tried with G1 GC. Only
difference is that, In case of CMS, even after starting solr, cpu goes to
80%, where as in G1, after started solr it's around 10% and when load comes
( around 100 to 200 docs  in 5 mins ) it's goes 90% ( in both CMS and G1 )

When we tried to profile the solr process, we found that merging is keep on
happening. 

This spikes of CPU and memory, we are seeing only after moving to 6.5.1. Is
that stable version ? moving to latest stable will solve this issue or we
miss something wrt configurations ? Do we need to change the solr default
config ? 


Advice... 




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Default Index config

[Shawn Heisey]

On 3/26/2018 10:45 AM, mganeshs wrote:
> I haven't changed the solr config wrt index config, which means it's all
> commented in the solrconfig.xml.
>
> It's something like what I pasted before. But I would like to know whats the
> default value of each of this.

Default value of *what* exactly?  There are a LOT of possible settings
you could be asking about, it'll take too much time to research them all
and give you every possible default.

I can't tell what you're referring to when you mention "what I pasted
before."  Later in your message I do see a mostly empty  tag.

If you're after defaults for indexConfig, the comments in Solr's example
config do a pretty good job of covering defaults.  I think that each
commented config section *has* the default setting already in it.  In a
download of the latest version, look for
server/solr/configsets/_default/conf/solrconfig.xml.

> Coz.. after loading to 6.5.1 and our document size also crossed 5GB in each
> of our collection. Now update of document is taking time. So would like to
> know whether we need to change any default configurations.

5GB for a *single* document?  That sounds like a document that probably
shouldn't get indexed, because it might cause all sorts of problems.

If 5GB is the size of the entire index, then I can say that while this
is not exactly a *small* index, there are a LOT of indexes in the wild
that are MUCH bigger.  I have such indexes in my Solr installs, and
while the performance of those indexes isn't lightning fast, it's pretty
good.

As your index size increases, Solr is going to slow down.  That's just
how things work and cannot be changed.  If the slowdown is extreme for a
small increase in size, that usually means that the system doesn't have
enough of some resource, usually memory.  Additional hardware or
hardware upgrades may be required.

If you describe what you are seeing, and how that differs from what you
EXPECT to be seeing, then we might be able to get somewhere.  More
information may be required to help.  If that's the case, we will ask
for additional information.

Thanks,
Shawn