Re: Solr Swap space
Thank you for this information! I am working on a client PC where Solr Admin showed 11.64 GB/13.03 GB Swap Space, and was losing my mind trying to track down the issue! -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr Swap space
Cool, Thanks, Shawn. I was also looking the swapiness and it is set to 60. Will try this out and let you know. Thanks, again. On Thu, Feb 22, 2018 at 10:55 AM, Shawn Heisey wrote: > On 2/21/2018 7:58 PM, Susheel Kumar wrote: > >> Below output for prod machine based on the steps you described. Please >> take a look. The solr searches are returning fine and no issue with >> performance but since last 4 months swap space started going up. After >> restart, it comes down to zero and then few weeks, it utilization reaches >> to 40-50% and thus requires restart of solr process. >> > > I bet that if you run this command, it will show you a value of 60: > > cat /proc/sys/vm/swappiness > > This makes the OS very aggressive about using swap, even when there is > absolutely no need for it to do so. > > If you type the following series of commands, it should fix the problem > and prevent it from happening again until you reboot the system: > > echo "0" > /proc/sys/vm/swappiness > swapoff -a > swapon -a > > Note that when the swapoff command runs, it will force the OS to read all > the swapped data back into memory. It will take several minutes for this > to occur, because it must read nearly a gigabyte of data and figure out how > to put it back in memory. Both of the command outputs you included say that > there is over 20GB of free memory. So I do not anticipate the system > having problems from running these commands. It will slow the machine down > temporarily, though -- so only do it during a quiet time for your Solr > install. > > To make this setting survive a reboot, find the sysctl.conf file somewhere > in your /etc directory and add this line to it: > > vm.swappiness = 0 > > This setting does not completely disable swap. If the system finds itself > with real memory pressure and actually does NEED to use swap, it still will > ... it just won't swap anything out before it's actually required. > > I do not think the behavior you are seeing is actually causing problems, > based on your system load and CPU usage. But what I've shared should fix > it for you. > > Thanks, > Shawn > >
Re: Solr Swap space
On 2/21/2018 7:58 PM, Susheel Kumar wrote: Below output for prod machine based on the steps you described. Please take a look. The solr searches are returning fine and no issue with performance but since last 4 months swap space started going up. After restart, it comes down to zero and then few weeks, it utilization reaches to 40-50% and thus requires restart of solr process. I bet that if you run this command, it will show you a value of 60: cat /proc/sys/vm/swappiness This makes the OS very aggressive about using swap, even when there is absolutely no need for it to do so. If you type the following series of commands, it should fix the problem and prevent it from happening again until you reboot the system: echo "0" > /proc/sys/vm/swappiness swapoff -a swapon -a Note that when the swapoff command runs, it will force the OS to read all the swapped data back into memory. It will take several minutes for this to occur, because it must read nearly a gigabyte of data and figure out how to put it back in memory. Both of the command outputs you included say that there is over 20GB of free memory. So I do not anticipate the system having problems from running these commands. It will slow the machine down temporarily, though -- so only do it during a quiet time for your Solr install. To make this setting survive a reboot, find the sysctl.conf file somewhere in your /etc directory and add this line to it: vm.swappiness = 0 This setting does not completely disable swap. If the system finds itself with real memory pressure and actually does NEED to use swap, it still will ... it just won't swap anything out before it's actually required. I do not think the behavior you are seeing is actually causing problems, based on your system load and CPU usage. But what I've shared should fix it for you. Thanks, Shawn
Re: Solr Swap space
Hi Shawn, Below output for prod machine based on the steps you described. Please take a look. The solr searches are returning fine and no issue with performance but since last 4 months swap space started going up. After restart, it comes down to zero and then few weeks, it utilization reaches to 40-50% and thus requires restart of solr process. top - 21:31:37 up 116 days, 20:09, 3 users, load average: 0.24, 0.24, 0.16 Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.4 us, 0.0 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65976996 total, 43846008 used, 22130988 free, 84 buffers KiB Swap: 2097148 total, 942228 used, 1154920 free. 39754116 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND SWAP 20327 solr 20 0 53.791g 0.021t 0.017t S 10.60 33.61 5109:53 java 547524 481 root20 0 267688 153312 9012 S 0.662 0.232 47:44.63 splunkd 34816 1293 root 20 0 94676 5536 5168 S 0.331 0.008 99:01.45 vmtoolsd 932 2699 root 20 0 21212 2740 2112 S 0.331 0.004 6:17.38 cvfwd 236 17706 kumarsu 20 0 46936 7600 5472 R 0.331 0.012 0:00.18 top 0 1 root 20 0 37532 5360 3896 S 0.000 0.008 6:07.28 systemd 248 2 root 20 0 0 0 0 S 0.000 0.000 0:04.66 kthreadd 0 3 root 20 0 0 0 0 S 0.000 0.000 0:38.21 ksoftirqd/0 0 5 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/0:0H 0 7 root 20 0 0 0 0 S 0.000 0.000 115:56.38 rcu_sched 0 8 root 20 0 0 0 0 S 0.000 0.000 0:00.00 rcu_bh 0 9 root rt 0 0 0 0 S 0.000 0.000 0:28.61 migration/0 0 10 root rt 0 0 0 0 S 0.000 0.000 0:24.68 watchdog/0 0 11 root rt 0 0 0 0 S 0.000 0.000 0:23.78 watchdog/1 0 12 root rt 0 0 0 0 S 0.000 0.000 0:29.63 migration/1 0 13 root 20 0 0 0 0 S 0.000 0.000 1:49.79 ksoftirqd/1 0 15 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/1:0H 0 16 root rt 0 0 0 0 S 0.000 0.000 0:24.00 watchdog/2 0 17 root rt 0 0 0 0 S 0.000 0.000 0:28.67 migration/2 0 18 root 20 0 0 0 0 S 0.000 0.000 0:26.13 ksoftirqd/2 0 20 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/2:0H 0 21 root rt 0 0 0 0 S 0.000 0.000 0:26.61 watchdog/3 0 22 root rt 0 0 0 0 S 0.000 0.000 0:27.92 migration/3 0 23 root 20 0 0 0 0 S 0.000 0.000 0:23.08 ksoftirqd/3 0 25 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/3:0H 0 26 root rt 0 0 0 0 S 0.000 0.000 0:28.02 watchdog/4 0 27 root rt 0 0 0 0 S 0.000 0.000 0:28.01 migration/4 0 28 root 20 0 0 0 0 S 0.000 0.000 0:51.07 ksoftirqd/4 0 31 root rt 0 0 0 0 S 0.000 0.000 0:25.29 watchdog/5 0 32 root rt 0 0 0 0 S 0.000 0.000 0:31.65 migration/5 0 33 root 20 0 0 0 0 S 0.000 0.000 0:22.52 ksoftirqd/5 0 35 root 0 -20 0 0 0 S 0.000 0.000 0:00.00 kworker/5:0H 0 36 root rt 0 0 0 0 S 0.000 0.000 0:25.90 watchdog/6 0 37 root rt 0 0 0 0 S 0.000 0.000 0:27.94 migration/6 0 38 root 20 0 0 0 0 S 0.000 0.000 1:07.91 ksoftirqd/6 0 40 root 0 -20 0 0 0
Re: Solr Swap space
On 2/21/2018 1:30 PM, Susheel Kumar wrote: > I did go thru your posts on swap usage http://lucene.472066.n3. > nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is > also similar. Below is top output from our prod and performance test > machine and as you can see the swap utilization on Prod machine is 44% > while on test machines it is zero. Are those "top" outputs sorted by the default (CPU usage) or by memory? To make any useful determination, it needs to be by memory. Press shift-M to sort by memory if your top program supports that key. Also, the list only shows the first few processes. More of the list needs to be visible. The system load on the second "top" output is quite low, and doesn't show much used CPU percentage. So it looks like the system is not actually suffering due to the swap usage, which probably means that it is not actively swapping. The machine has plenty of memory available -- even though almost all of the memory is allocated, the vast majority of what's allocated is in the "cached" state -- used by the OS disk cache. The OS will instantly give up this memory if a program requests it. I've learned how to use top to show which processes are using swap. What I've described below should work on recent version of gnu top. If the top is from another software provider, it may not support this. http://northernmost.org/blog/swap-usage-5-years-later/ These steps are not precisely as described in that blog post: Run top, press f, press p, press space, then press the right angle bracket (>) key three times. If top is running with default settings, these keypresses should enable the SWAP column and move the sort to that column. The list should be sorted by swap usage. If a .toprc file exists in your home directory, then the program may be running with very different settings than default, and these keypresses might not work as expected. Thanks, Shawn
Re: Solr Swap space
Hello Shawn, I did go thru your posts on swap usage http://lucene.472066.n3. nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is also similar. Below is top output from our prod and performance test machine and as you can see the swap utilization on Prod machine is 44% while on test machines it is zero. I haven't been able to figure out what/where exactly is the problem. Both test and prod machines, in terms of Java version are same 1.8.0_91. What could help to further understand and debug the issue? Thanks, Susheel *Performance test machine * top - 15:03:52 up 37 days, 5:49, 1 user, load average: 0.06, 0.09, 0.04 Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65977076 total, 60079356 used, 5897720 free, 1080 buffers KiB Swap: 2097148 total,0 used, 2097148 free. 56416304 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 433 solr 20 071.813g 3.484g 311548 S 1.667 5.538 105:37.51 java 1 root20 0 37504 5672 4020 S 0.000 0.009 3:43.54 systemd 2 root20 0 0 0 0 S 0.000 0.000 0:04.91 kthreadd ... ... *Prod machine* op - 15:10:20 up 116 days, 13:47, 3 users, load average: 0.16, 0.20, 0.18 Tasks: 200 total, 1 running, 199 sleeping, 0 stopped, 0 zombie %Cpu(s): 2.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65976996 total, 65460256 used, 516740 free, 84 buffers KiB Swap: 2097148 total, *942288* used, 1154860 free. 61361628 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 20327 solr 20 0 53.792g 0.021t 0.017t S 15.95 33.42 5051:07 java 481 root 20 0 267688 152256 9012 S 0.332 0.231 46:47.60 splunkd 1 root 20 0 37532 5360 3896 S 0.000 0.008 6:06.47 systemd ... ... On Fri, Feb 9, 2018 at 5:44 PM, Emir Arnautović < emir.arnauto...@sematext.com> wrote: > No worries, I don't mind being confused with Erick ;) > > Emir > > On Feb 9, 2018 9:16 PM, "Susheel Kumar" wrote: > > > Sorry, I meant Emir. > > > > On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar > > wrote: > > > > > Thanks, Shawn, Eric. I see that same using swapon -s. Looks like > during > > > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6) > > > > > > Our 6.0 instance has been running since 1+ year but recently our monit > > > started reporting swap usage above 30% and Solr dashboard showing the > > > same. I haven't been able to find what causes / why Solr is using the > > swap > > > space. The index size is well within memory size to fit and we have > > double > > > the index size on our performance test machines but there is no usage > of > > > swap space. > > > > > > Thanks, > > > Susheel > > > > > > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey > > wrote: > > > > > >> On 2/7/2018 12:01 PM, Susheel Kumar wrote: > > >> > > >>> Just trying to find where do we set swap space available to Solr > > >>> process. I > > >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances > its > > >>> set > > >>> to 16GB. > > >>> > > >> > > >> Solr has absolutely no involvement or control over swap space. > Neither > > >> does Java. This is a function of your operating system's memory > > >> management, and is typically set up when you first install your OS. > > >> > > >> https://www.linux.com/news/all-about-linux-swap-space > > >> https://en.wikipedia.org/wiki/Paging#Windows_NT > > >> > > >> If your system is using swap space, it's a strong indication that you > > >> don't have enough memory installed. If any of the memory that Solr > > uses is > > >> swapped out to disk, Solr performance is going to be REALLY bad. > > >> > > >> Thanks, > > >> Shawn > > >> > > > > > > > > >
Re: Solr Swap space
No worries, I don't mind being confused with Erick ;) Emir On Feb 9, 2018 9:16 PM, "Susheel Kumar" wrote: > Sorry, I meant Emir. > > On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar > wrote: > > > Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during > > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6) > > > > Our 6.0 instance has been running since 1+ year but recently our monit > > started reporting swap usage above 30% and Solr dashboard showing the > > same. I haven't been able to find what causes / why Solr is using the > swap > > space. The index size is well within memory size to fit and we have > double > > the index size on our performance test machines but there is no usage of > > swap space. > > > > Thanks, > > Susheel > > > > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey > wrote: > > > >> On 2/7/2018 12:01 PM, Susheel Kumar wrote: > >> > >>> Just trying to find where do we set swap space available to Solr > >>> process. I > >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its > >>> set > >>> to 16GB. > >>> > >> > >> Solr has absolutely no involvement or control over swap space. Neither > >> does Java. This is a function of your operating system's memory > >> management, and is typically set up when you first install your OS. > >> > >> https://www.linux.com/news/all-about-linux-swap-space > >> https://en.wikipedia.org/wiki/Paging#Windows_NT > >> > >> If your system is using swap space, it's a strong indication that you > >> don't have enough memory installed. If any of the memory that Solr > uses is > >> swapped out to disk, Solr performance is going to be REALLY bad. > >> > >> Thanks, > >> Shawn > >> > > > > >
Re: Solr Swap space
Sorry, I meant Emir. On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar wrote: > Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6) > > Our 6.0 instance has been running since 1+ year but recently our monit > started reporting swap usage above 30% and Solr dashboard showing the > same. I haven't been able to find what causes / why Solr is using the swap > space. The index size is well within memory size to fit and we have double > the index size on our performance test machines but there is no usage of > swap space. > > Thanks, > Susheel > > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey wrote: > >> On 2/7/2018 12:01 PM, Susheel Kumar wrote: >> >>> Just trying to find where do we set swap space available to Solr >>> process. I >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its >>> set >>> to 16GB. >>> >> >> Solr has absolutely no involvement or control over swap space. Neither >> does Java. This is a function of your operating system's memory >> management, and is typically set up when you first install your OS. >> >> https://www.linux.com/news/all-about-linux-swap-space >> https://en.wikipedia.org/wiki/Paging#Windows_NT >> >> If your system is using swap space, it's a strong indication that you >> don't have enough memory installed. If any of the memory that Solr uses is >> swapped out to disk, Solr performance is going to be REALLY bad. >> >> Thanks, >> Shawn >> > >
Re: Solr Swap space
Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6) Our 6.0 instance has been running since 1+ year but recently our monit started reporting swap usage above 30% and Solr dashboard showing the same. I haven't been able to find what causes / why Solr is using the swap space. The index size is well within memory size to fit and we have double the index size on our performance test machines but there is no usage of swap space. Thanks, Susheel On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey wrote: > On 2/7/2018 12:01 PM, Susheel Kumar wrote: > >> Just trying to find where do we set swap space available to Solr process. >> I >> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set >> to 16GB. >> > > Solr has absolutely no involvement or control over swap space. Neither > does Java. This is a function of your operating system's memory > management, and is typically set up when you first install your OS. > > https://www.linux.com/news/all-about-linux-swap-space > https://en.wikipedia.org/wiki/Paging#Windows_NT > > If your system is using swap space, it's a strong indication that you > don't have enough memory installed. If any of the memory that Solr uses is > swapped out to disk, Solr performance is going to be REALLY bad. > > Thanks, > Shawn >
Re: Solr Swap space
On 2/7/2018 12:01 PM, Susheel Kumar wrote: Just trying to find where do we set swap space available to Solr process. I see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set to 16GB. Solr has absolutely no involvement or control over swap space. Neither does Java. This is a function of your operating system's memory management, and is typically set up when you first install your OS. https://www.linux.com/news/all-about-linux-swap-space https://en.wikipedia.org/wiki/Paging#Windows_NT If your system is using swap space, it's a strong indication that you don't have enough memory installed. If any of the memory that Solr uses is swapped out to disk, Solr performance is going to be REALLY bad. Thanks, Shawn
Re: Solr Swap space
Hi Susheel, Swap space is OS thing, not Solr thing. You should see how to disable swap space or at least set swappiness to some low number on your OS. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 7 Feb 2018, at 20:01, Susheel Kumar wrote: > > Hello, > > Just trying to find where do we set swap space available to Solr process. I > see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set > to 16GB. > > Thanks, > Susheel
