Linux automatically uses free memory as cache.  It's not swap.

http://www.tldp.org/LDP/lki/lki-4.html

On Sat, Oct 8, 2016 at 11:12 AM Vladimir Yudovin <vla...@winguzone.com>
wrote:

> Sorry, I don't catch something. What page (memory) cache can exist if
> there is no swap file.
> Where are those page written/read?
>
>
> Best regards, Vladimir Yudovin,
>
>
> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud Cassandra on
> Azure and SoftLayer.Launch your cluster in minutes.*
>
>
> ---- On Sat, 08 Oct 2016 14:09:50 -0400 *Ariel Weisberg<ar...@weisberg.ws
> <ar...@weisberg.ws>>* wrote ----
>
> Hi,
>
> Nope I mean page cache. Linux doesn't call the cache it maintains using
> free memory a file cache. It uses free (and some of the time not so free!)
> memory to buffer writes and to cache recently written/read data.
>
> http://www.tldp.org/LDP/lki/lki-4.html
>
> When Linux decides it needs free memory it can either evict stuff from the
> page cache, flush dirty pages and then evict, or swap anonymous memory out.
> When you disable swap you only disable the last behavior.
>
> Maybe we are talking at cross purposes? What I meant is that increasing
> the heap size to reduce GC frequency is a legitimate thing to do and it
> does have an impact on the performance of the page cache even if you have
> swap disabled?
>
> Ariel
>
>
> On Sat, Oct 8, 2016, at 01:54 PM, Vladimir Yudovin wrote:
>
> >Page cache is data pending flush to disk and data cached from disk.
>
> Do you mean file cache?
>
>
> Best regards, Vladimir Yudovin,
>
>
> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud Cassandra on
> Azure and SoftLayer.Launch your cluster in minutes.*
>
>
> ---- On Sat, 08 Oct 2016 13:40:19 -0400 *Ariel Weisberg
> <ar...@weisberg.ws <ar...@weisberg.ws>>* wrote ----
>
> Hi,
>
> Page cache is in use even if you disable swap. Swap is anonymous memory,
> and whatever else the Linux kernel supports paging out. Page cache is data
> pending flush to disk and data cached from disk.
>
> Given how bad the GC pauses are in C* I think it's not the high pole in
> the tent. Until key things are off heap and C* can run with CMS and get 10
> millisecond GCs all day long.
>
> You can go through tuning and hardware selection try to get more
> consistent IO pauses and remove outliers as you mention and as a user I
> think this is your best bet. Generally it's either bad device or filesystem
> behavior if you get page faults taking more than 200 milliseconds O(G1 gc
> collection).
>
> I think a JVM change to allow safe points around memory mapped file access
> is really unlikely although I agree it would be great. I think the best
> hack around it is to code up your memory mapped file access into JNI
> methods and find some way to get that to work. Right now if you want to
> create a safe point a JNI method is the way to do it. The problem is that
> JNI methods and POJOs don't get along well.
>
> If you think about it the reason non-memory mapped IO works well is that
> it's all JNI methods so they don't impact time to safe point. I think there
> is a tradeoff between tolerance for outliers and performance.
>
> I don't know the state of the non-memory mapped path and how reliable that
> is. If it were reliable and I couldn't tolerate the outliers I would use
> that. I have to ask though, why are you not able to tolerate the outliers?
> If you are reading and writing at quorum how is this impacting you?
>
> Regards,
> Ariel
>
> On Sat, Oct 8, 2016, at 12:54 AM, Vladimir Yudovin wrote:
>
> Hi Josh,
>
> >Running with increased heap size would reduce GC frequency, at the cost
> of page cache.
>
> Actually it's recommended to run C* without virtual memory enabled. So if
> there is no enough memory JVM fails instead of blocking
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud Cassandra on
> Azure and SoftLayer.Launch your cluster in minutes.*
>
>
> ---- On Fri, 07 Oct 2016 21:06:24 -0400 *Josh Snyder<j...@code406.com
> <j...@code406.com>>* wrote ----
>
> Hello cassandra-users,
>
> I'm investigating an issue with JVMs taking a while to reach a safepoint.
> I'd
> like the list's input on confirming my hypothesis and finding mitigations.
>
> My hypothesis is that slow block devices are causing Cassandra's JVM to
> pause
> completely while attempting to reach a safepoint.
>
> Background:
>
> Hotspot occasionally performs maintenance tasks that necessitate stopping
> all
> of its threads. Threads running JITed code occasionally read from a given
> safepoint page. If Hotspot has initiated a safepoint, reading from that
> page
> essentially catapults the thread into purgatory until the safepoint
> completes
> (the mechanism behind this is pretty cool). Threads performing syscalls or
> executing native code do this check upon their return into the JVM.
>
> In this way, during the safepoint Hotspot can be sure that all of its
> threads
> are either patiently waiting for safepoint completion or in a system call.
>
> Cassandra makes heavy use of mmapped reads in normal operation. When doing
> mmapped reads, the JVM executes userspace code to effect a read from a
> file. On
> the fast path (when the page needed is already mapped into the process),
> this
> instruction is very fast. When the page is not cached, the CPU triggers a
> page
> fault and asks the OS to go fetch the page. The JVM doesn't even realize
> that
> anything interesting is happening: to it, the thread is just executing a
> mov
> instruction that happens to take a while.
>
> The OS, meanwhile, puts the thread in question in the D state (assuming
> Linux,
> here) and goes off to find the desired page. This may take microseconds,
> this
> may take milliseconds, or it may take seconds (or longer). When I/O occurs
> while the JVM is trying to enter a safepoint, every thread has to wait for
> the
> laggard I/O to complete.
>
> If you log safepoints with the right options [1], you can see these
> occurrences
> in the JVM output:
>
> > # SafepointSynchronize::begin: Timeout detected:
> > # SafepointSynchronize::begin: Timed out while spinning to reach a
> safepoint.
> > # SafepointSynchronize::begin: Threads which did not reach the
> safepoint:
> > # "SharedPool-Worker-5" #468 daemon prio=5 os_prio=0
> tid=0x00007f8785bb1f30 nid=0x4e14 runnable [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> >
> > # SafepointSynchronize::begin: (End of list)
> > vmop [threads: total initially_running wait_to_block] [time: spin block
> sync cleanup vmop] page_trap_count
> > 58099.941: G1IncCollectionPause [ 447 1 1 ] [ 3304 0 3305 1 190 ] 1
>
> If that safepoint happens to be a garbage collection (which this one was),
> you
> can also see it in GC logs:
>
> > 2016-10-07T13:19:50.029+0000: 58103.440: Total time for which
> application threads were stopped: 3.4971808 seconds, Stopping threads took:
> 3.3050644 seconds
>
> In this way, JVM safepoints become a powerful weapon for transmuting a
> single
> thread's slow I/O into the entire JVM's lockup.
>
> Does all of the above sound correct?
>
> Mitigations:
>
> 1) don't tolerate block devices that are slow
>
> This is easy in theory, and only somewhat difficult in practice. Tools
> like
> perf and iosnoop [2] can do pretty good jobs of letting you know when a
> block
> device is slow.
>
> It is sad, though, because this makes running Cassandra on mixed hardware
> (e.g.
> fast SSD and slow disks in a JBOD) quite unappetizing.
>
> 2) have fewer safepoints
>
> Two of the biggest sources of safepoints are garbage collection and
> revocation
> of biased locks. Evidence points toward biased locking being unhelpful for
> Cassandra's purposes, so turning it off (-XX:-UseBiasedLocking) is a quick
> way
> to eliminate one source of safepoints.
>
> Garbage collection, on the other hand, is unavoidable. Running with
> increased
> heap size would reduce GC frequency, at the cost of page cache. But
> sacrificing
> page cache would increase page fault frequency, which is another thing
> we're
> trying to avoid! I don't view this as a serious option.
>
> 3) use a different IO strategy
>
> Looking at the Cassandra source code, there appears to be an
> un(der)documented
> configuration parameter called disk_access_mode. It appears that changing
> this
> to 'standard' would switch to using pread() and pwrite() for I/O, instead
> of
> mmap. I imagine there would be a throughput penalty here for the case when
> pages are in the disk cache.
>
> Is this a serious option? It seems far too underdocumented to be thought
> of as
> a contender.
>
> 4) modify the JVM
>
> This is a longer term option. For the purposes of safepoints, perhaps the
> JVM
> could treat reads from an mmapped file in the same way it treats threads
> that
> are running JNI code. That is, the safepoint will proceed even though the
> reading thread has not "joined in". Upon finishing its mmapped read, the
> reading thread would test the safepoint page (check whether a safepoint is
> in
> progress, in other words).
>
> Conclusion:
>
> I don't imagine there's an easy solution here. I plan to go ahead with
> mitigation #1: "don't tolerate block devices that are slow", but I'd
> appreciate
> any approach that doesn't require my hardware to be flawless all the time.
>
> Josh
>
> [1] -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=100
> -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1
> [2] https://github.com/brendangregg/perf-tools/blob/master/iosnoop
>
>
>
>
>

Reply via email to