Re: Sorl shards: very sensitive to swap space usage !?

[Toke Eskildsen]

On Mon, 2016-11-14 at 16:29 -0800, Chetas Joshi wrote:
> Hi Toke, can you explain exactly what you mean by "the aggressive IO
> for the memory mapping caused the kernel to start swapping parts of
> the JVM heap to get better caching of storage data"?

I am not sure what you are asking for. I'll try adding more details:


Our machine(s) which ran into the swap problem had 256GB of physical
memory, with some 50GB+ free for caching, but handled multi terabytes
of index. So the free memory for memory mapping (aka disk cache) was
around 1% if the index size. With 25 active shards on the machine, each
search request resulted in a lot of IO to map memory from index data to
physical memory.

Solr JVMs on the machine did not do a lot of garbage collection. Partly
because of low query rate, partly because of some internal hacks.

So we had a machine with very heavy memory mapping and not-too-active
JVM heaps.

The principle behind swap is to store infrequently used memory in on
slower storage. This is where I guess that the kernel guessed that
freeing more memory for mapping, by pushing relatively stale parts of
the JVM heaps onto swap, would result in overall better performance.

- Toke Eskildsen, State and University Library, Denmark

Re: Sorl shards: very sensitive to swap space usage !?

[Chetas Joshi]

Thanks everyone!
The discussion is really helpful.

Hi Toke, can you explain exactly what you mean by "the aggressive IO for
the memory mapping caused the kernel to start swapping parts of the JVM
heap to get better caching of storage data"?
Which JVM are you talking about? Solr shard? I have other services running
on the same host as well.

Thanks!

On Fri, Nov 11, 2016 at 7:32 AM, Shawn Heisey  wrote:

> On 11/11/2016 6:46 AM, Toke Eskildsen wrote:
> > but on two occasions I have
> > experienced heavy swapping with multiple gigabytes free for disk
> > cache. In both cases, the cache-to-index size was fairly low (let's
> > say < 10%). My guess (I don't know the intrinsics of memory mapping
> > vs. swapping) is that the aggressive IO for the memory mapping caused
> > the kernel to start swapping parts of the JVM heap to get better
> > caching of storage data. Yes, with terrible performance as a result.
>
> That's really weird, and sounds like a broken operating system.  I've
> had other issues with swap, but in those cases, free memory was actually
> near zero, and it sounds like your situation was not the same.  So the
> OP here might be having similar problems even if nothing's
> misconfigured.  If so, your solution will probably help them.
>
> > No matter the cause, the swapping problems were "solved" by
> > effectively disabling the swap (swappiness 0).
>
> Solr certainly doesn't need (or even want) swap, if the machine is sized
> right.  I've read some things saying that Linux doesn't behave correctly
> if you completely get rid of all swap, but setting swappiness to zero
> sounds like a good option.  The OS would still utilize swap if it
> actually ran out of physical memory, so you don't lose the safety valve
> that swap normally provides.
>
> Thanks,
> Shawn
>
>

Re: Sorl shards: very sensitive to swap space usage !?

[Shawn Heisey]

On 11/11/2016 6:46 AM, Toke Eskildsen wrote:
> but on two occasions I have
> experienced heavy swapping with multiple gigabytes free for disk
> cache. In both cases, the cache-to-index size was fairly low (let's
> say < 10%). My guess (I don't know the intrinsics of memory mapping
> vs. swapping) is that the aggressive IO for the memory mapping caused
> the kernel to start swapping parts of the JVM heap to get better
> caching of storage data. Yes, with terrible performance as a result. 

That's really weird, and sounds like a broken operating system.  I've
had other issues with swap, but in those cases, free memory was actually
near zero, and it sounds like your situation was not the same.  So the
OP here might be having similar problems even if nothing's
misconfigured.  If so, your solution will probably help them.

> No matter the cause, the swapping problems were "solved" by
> effectively disabling the swap (swappiness 0).

Solr certainly doesn't need (or even want) swap, if the machine is sized
right.  I've read some things saying that Linux doesn't behave correctly
if you completely get rid of all swap, but setting swappiness to zero
sounds like a good option.  The OS would still utilize swap if it
actually ran out of physical memory, so you don't lose the safety valve
that swap normally provides.

Thanks,
Shawn

Re: Sorl shards: very sensitive to swap space usage !?

[Toke Eskildsen]

On Thu, 2016-11-10 at 16:42 -0700, Shawn Heisey wrote:
> If the machine that Solr is installed on is using swap, that means
> you're having serious problems, and your performance will be
> TERRIBLE. 

Agreed so far.

> This kind of problem cannot be caused by Solr if it is properly
> configured for the machine it's running on.

That is practically a tautology.


Most of the Solr setups I have worked with has been behaving as one
would hope with regards to swap, but on two occasions I have
experienced heavy swapping with multiple gigabytes free for disk cache.
In both cases, the cache-to-index size was fairly low (let's say <
10%). My guess (I don't know the intrinsics of memory mapping vs.
swapping) is that the aggressive IO for the memory mapping caused the
kernel to start swapping parts of the JVM heap to get better caching of
storage data. Yes, with terrible performance as a result.

No matter the cause, the swapping problems were "solved" by effectively
disabling the swap (swappiness 0). We did try with very conservative
swapping first (swappiness 5 or something like that), but that did not
work. Although that meant less free memory for disk caching, as nothing
were no longer swapped, it solved our performance problems.


Disabling swapping is easy to try, so I suggest doing just that.

- Toke Eskildsen, State and University Library, Denmark

Re: Sorl shards: very sensitive to swap space usage !?

[Kevin Risden]

Agreed with what Shawn and Erick said.

If you don't see anything in the Solr logs and your servers are swapping a
lot, this could mean the Linux OOM killer is killing the Solr process (and
maybe others). There is usually a log of this depending on your Linux
distribution.

Kevin Risden

On Thu, Nov 10, 2016 at 6:42 PM, Shawn Heisey  wrote:

> On 11/10/2016 3:20 PM, Chetas Joshi wrote:
> > I have a SolrCloud (Solr 5.5.0) of 50 nodes. The JVM heap memory usage
> > of my solr shards is never more than 50% of the total heap. However,
> > the hosts on which my solr shards are deployed often run into 99% swap
> > space issue. This causes the solr shards go down. Why solr shards are
> > so sensitive to the swap space usage? The JVM heap is more than enough
> > so the shards should never require the swap space. What could be the
> > reason? Where can find the reason why the solr shards go down. I don't
> > see anything on the solr logs.
>
> If the machine that Solr is installed on is using swap, that means
> you're having serious problems, and your performance will be TERRIBLE.
> This kind of problem cannot be caused by Solr if it is properly
> configured for the machine it's running on.
>
> Solr is a Java program.  That means its memory usage is limited to the
> Java heap, plus a little bit for Java itself, and absolutely cannot go
> any higher.  If the Java heap is set too large, then the operating
> system might utilize swap to meet Java's memory demands.  The solution
> is to set your Java heap to a value that's significantly smaller than
> the amount of available physical memory.  Setting the heap to a value
> that's close to (or more than) the amount of physical memory, is a
> recipe for very bad performance.
>
> You need to also limit the memory usage of other software installed on
> the machine, or you might run into a situation where swap is required
> that is not Solr's fault.
>
> Thanks,
> Shawn
>
>

Re: Sorl shards: very sensitive to swap space usage !?

[Shawn Heisey]

On 11/10/2016 3:20 PM, Chetas Joshi wrote:
> I have a SolrCloud (Solr 5.5.0) of 50 nodes. The JVM heap memory usage
> of my solr shards is never more than 50% of the total heap. However,
> the hosts on which my solr shards are deployed often run into 99% swap
> space issue. This causes the solr shards go down. Why solr shards are
> so sensitive to the swap space usage? The JVM heap is more than enough
> so the shards should never require the swap space. What could be the
> reason? Where can find the reason why the solr shards go down. I don't
> see anything on the solr logs.

If the machine that Solr is installed on is using swap, that means
you're having serious problems, and your performance will be TERRIBLE. 
This kind of problem cannot be caused by Solr if it is properly
configured for the machine it's running on.

Solr is a Java program.  That means its memory usage is limited to the
Java heap, plus a little bit for Java itself, and absolutely cannot go
any higher.  If the Java heap is set too large, then the operating
system might utilize swap to meet Java's memory demands.  The solution
is to set your Java heap to a value that's significantly smaller than
the amount of available physical memory.  Setting the heap to a value
that's close to (or more than) the amount of physical memory, is a
recipe for very bad performance.

You need to also limit the memory usage of other software installed on
the machine, or you might run into a situation where swap is required
that is not Solr's fault.

Thanks,
Shawn

Re: Sorl shards: very sensitive to swap space usage !?

[Erick Erickson]

Lots of data from the index is kept in MMapDirectory space, which is
mapped to the OS's memory, see Uwe's excellent explanation here:

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Best,
Erick

On Thu, Nov 10, 2016 at 2:20 PM, Chetas Joshi  wrote:
> Hi,
>
> I have a SolrCloud (Solr 5.5.0) of 50 nodes. The JVM heap memory usage of
> my solr shards is never more than 50% of the total heap. However, the hosts
> on which my solr shards are deployed often run into 99% swap space issue.
> This causes the solr shards go down. Why solr shards are so sensitive to
> the swap space usage? The JVM heap is more than enough so the shards should
> never require the swap space. What could be the reason? Where can find the
> reason why the solr shards go down. I don't see anything on the solr logs.
>
> Thanks!

Sorl shards: very sensitive to swap space usage !?

[Chetas Joshi]

Hi,

I have a SolrCloud (Solr 5.5.0) of 50 nodes. The JVM heap memory usage of
my solr shards is never more than 50% of the total heap. However, the hosts
on which my solr shards are deployed often run into 99% swap space issue.
This causes the solr shards go down. Why solr shards are so sensitive to
the swap space usage? The JVM heap is more than enough so the shards should
never require the swap space. What could be the reason? Where can find the
reason why the solr shards go down. I don't see anything on the solr logs.

Thanks!